[cloud][docker] use the private docker namespace #28286

v1v · 2021-10-06T15:19:16Z

What does this PR do?

Use the beats-ci namespace ONLY for the elastic-agent-cloud docker images.

Why is it important?

These images are not made to be consumed outside, so the UR suggested to use the beats-ci similar to what Elasticsearch/Kibana do

Test

Packaging

https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/packaging/branches/PR-28286/runs/3/nodes/246/log/?start=0

https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/packaging/branches/PR-28286/runs/3/nodes/4110/log/?start=0

mergify · 2021-10-06T15:19:48Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

v1v · 2021-10-06T15:20:16Z

/package

elasticmachine · 2021-10-06T16:26:45Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: Aborted from #2
Start Time: 2021-10-06T15:19:33.403+0000
Duration: 67 min 38 sec
Commit: e016690

Test stats 🧪

Test	Results
Failed	0
Passed	18713
Skipped	1412
Total	20125

Steps errors

Expand to view the steps failures

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

Took 0 min 2 sec . View more details here
Description: gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1

`x-pack/filebeat-unitTest - mage build unitTest`

Took 5 min 57 sec . View more details here
Description: mage build unitTest

`x-pack/metricbeat-unitTest - mage build unitTest`

Took 5 min 36 sec . View more details here
Description: mage build unitTest

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

Took 0 min 4 sec . View more details here
Description: gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

v1v · 2021-10-07T07:52:45Z

Existing packaging failures are not related to this change but some timeout

But it worked in the main pipeline

jlind23 · 2021-10-07T08:16:12Z

@v1v could you try a rebuild to check if timeouts still occur?

v1v · 2021-10-07T08:18:51Z

@v1v could you try a rebuild to check if timeouts still occur?

That's already in progress, thanks for double checking it 💯

jlind23 · 2021-10-07T09:08:49Z

@v1v do you know why the packaging task has not been relaunched?

v1v · 2021-10-07T09:56:42Z

@v1v do you know why the packaging task has not been relaunched?

Yes. I triggered the Packaging pipeline manually rather than the whole pipeline, the Packaging GitHub check gets updated when running the main pipeline, since I didn't want to run the main pipeline again, I triggered manually the Packaging pipeline (that's similar to add the comment /package).

Packaging build5 ran successfully.

v1v · 2021-10-07T10:05:55Z

This PR has successfully ran in the CI, see the below summary:

The main pipeline failed at the very end when running the packaging stage since it took over 4 hours.
A manual packaging was triggered manually afterwards (see packaging build)
The e2e testing passed (see e2e build)

Therefore, It's not needed to run everything again in the CI for the above-mentioned details.

jlind23 · 2021-10-07T12:55:31Z

I guess that you now need an administrator to "force" merge your PR right?

v1v · 2021-10-07T12:58:43Z

I guess that you now need an administrator to "force" merge your PR right?

I can do it, but I want to have an approval from the Beats team if possible or someone who has a deeper knowledge of the packaging system.

v1v · 2021-10-07T15:45:33Z

I'll merge this PR now

* upstream/master: (73 commits) Remove GCP support from Functionbeat (elastic#28253) Move labels and annotations under kubernetes.namespace. (elastic#27917) Update go release version 1.17.1 (elastic#27543) Osquerybeat: Runner and Fetcher unit tests (elastic#28290) Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) seccomp: allow clone3 syscall for x86 (elastic#28117) packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) [cloud][docker] use the private docker namespace (elastic#28286) Update aws-lambda-go library version to 1.13.3 (elastic#28236) Deprecate common.Float (elastic#28280) Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274) x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124) Explicitly pass http config to doppler consumer (elastic#28277) processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219) Resolved timestamp for defender atp (elastic#28272) [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191) [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165) Build cloud docker images for elastic-agent (elastic#28134) Upgrade k8s go-client library (elastic#28228) ...

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (#28236) * [cloud][docker] use the private docker namespace (#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (#28297) * seccomp: allow clone3 syscall for x86 (#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (#28253) * Fix build tags for Go 1.17 (#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (#28325) * Redis: remove deprecated fields (#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (#27878) partial fix for #27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (elastic#28236) * [cloud][docker] use the private docker namespace (elastic#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) * seccomp: allow clone3 syscall for x86 (elastic#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (elastic#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (elastic#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (elastic#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (elastic#28253) * Fix build tags for Go 1.17 (elastic#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (elastic#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325) * Redis: remove deprecated fields (elastic#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (elastic#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (elastic#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

v1v added 2 commits October 6, 2021 16:15

[cloud][docker] use the private docker namespace

fe5efd4

Add earlier

e016690

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 6, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 6, 2021

v1v requested review from jsoriano, blakerouse and a team October 6, 2021 15:21

v1v self-assigned this Oct 6, 2021

v1v added :Packaging automation Team:Automation Label for the Observability productivity team labels Oct 6, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 6, 2021

Customise docker namespace for Cloud

07d582e

cachedout approved these changes Oct 7, 2021

View reviewed changes

kuisathaverat approved these changes Oct 7, 2021

View reviewed changes

v1v merged commit 8f969e7 into elastic:master Oct 7, 2021

v1v deleted the feature/use-beats-ci branch October 7, 2021 15:45

newly12 pushed a commit to newly12/beats that referenced this pull request Oct 13, 2021

[cloud][docker] use the private docker namespace (elastic#28286)

80b9e28

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

[cloud][docker] use the private docker namespace (elastic#28286)

da4b943

v1v mentioned this pull request Mar 16, 2022

[7.17](backport #30795) ci: refactor pushDockerImages main pipeline #30846

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cloud][docker] use the private docker namespace #28286

[cloud][docker] use the private docker namespace #28286

v1v commented Oct 6, 2021 •

edited

Loading

mergify bot commented Oct 6, 2021

v1v commented Oct 6, 2021

elasticmachine commented Oct 6, 2021

Build stats

Test stats 🧪

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

`x-pack/filebeat-unitTest - mage build unitTest`

`x-pack/metricbeat-unitTest - mage build unitTest`

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

`Error signal`

v1v commented Oct 7, 2021 •

edited

Loading

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

v1v commented Oct 7, 2021

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

v1v commented Oct 7, 2021

[cloud][docker] use the private docker namespace #28286

[cloud][docker] use the private docker namespace #28286

Conversation

v1v commented Oct 6, 2021 • edited Loading

What does this PR do?

Why is it important?

Test

Packaging

mergify bot commented Oct 6, 2021

v1v commented Oct 6, 2021

elasticmachine commented Oct 6, 2021

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1

x-pack/filebeat-unitTest - mage build unitTest

x-pack/metricbeat-unitTest - mage build unitTest

gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1

Error signal

🤖 GitHub comments

v1v commented Oct 7, 2021 • edited Loading

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

v1v commented Oct 7, 2021

jlind23 commented Oct 7, 2021

v1v commented Oct 7, 2021

v1v commented Oct 7, 2021

v1v commented Oct 6, 2021 •

edited

Loading

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

`x-pack/filebeat-unitTest - mage build unitTest`

`x-pack/metricbeat-unitTest - mage build unitTest`

`gsutil -m -q cp -a public-read build/system-tests-*.tar.gz gs://beats-ci-temp/Beats/beats/PR-28286-1`

`Error signal`

v1v commented Oct 7, 2021 •

edited

Loading