Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: generate complete variant binaries properly #27609

Closed

Conversation

mdelapenya
Copy link
Contributor

@mdelapenya mdelapenya commented Aug 26, 2021

What does this PR do?

This PR is enriching the image name for the "complete" variant of the Elastic Agent when building it using mage package. It simply adds an extra spec that must be added to the complete variant. We are adding an additional build package for the Elastic Agent, exactly the same as for the UBI8 variant.

Given I'm not a Beats build expert, I'm copying what the ubi8 variant does, because it seems the variants keyword used in the past is not doing its job. Maybe @jsoriano or @kvch knows more about it.

Why is it important?

We realised that CI builds were generating incorrect binaries for the agent, as the elastic-agent docker image (TAR.GZ binary) was pushed to the internal GCP bucket only once, as described in #27608. Then, any consumer of those binaries was always receiving a +500MB file when downloading it from the bucket. An example of a consumer is the e2e-testing framework, which gets a binary for a merge commit or a PR in Beats project, and runs the E2E tests against those non-released delta binaries. We noticed the download times for the default artifact increased up to the double, also missing the complete TAR.GZ binary in the GCP bucket.

With this fix we expect the CI artifacts, which are super useful for bisecting errors, will be generated properly again.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Is there any other mage package combination that I missed?
  • Is the spec I generated complete?
  • Is there a variants key in the packages.yml file? How is it used, because without this fix it seemed not to work as expected?

How to test this PR locally

cd x-pack/elastic-agent
PLATFORMS="linux/amd64" mage package

After around 10 minutes... check for the binaries produced by the build system:

ls -l x-pack/elastic-agent/build/distributions | grep docker
-rw-r--r--  1 mdelapenya  staff  244359238 26 Aug 13:51 elastic-agent-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        175 26 Aug 13:51 elastic-agent-8.0.0-linux-amd64.docker.tar.gz.sha512
-rw-r--r--  1 mdelapenya  staff  614539725 26 Aug 13:55 elastic-agent-complete-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        184 26 Aug 13:56 elastic-agent-complete-8.0.0-linux-amd64.docker.tar.gz.sha512
-rw-r--r--  1 mdelapenya  staff  163226167 26 Aug 13:50 elastic-agent-ubi8-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        180 26 Aug 13:50 elastic-agent-ubi8-8.0.0-linux-amd64.docker.tar.gz.sha512

Related issues

@mdelapenya mdelapenya self-assigned this Aug 26, 2021
@mdelapenya mdelapenya added the Team:Automation Label for the Observability productivity team label Aug 26, 2021
@mdelapenya mdelapenya requested a review from a team August 26, 2021 12:58
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 26, 2021
@mdelapenya mdelapenya added Team:Elastic-Agent Label for the Agent team bug build-system Issue or change affecting Mage, Make, or build scripts. Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Aug 26, 2021
@mdelapenya mdelapenya marked this pull request as ready for review August 26, 2021 14:03
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@mdelapenya
Copy link
Contributor Author

BTW, complete artifacts being uploaded to the bucket for my PR: please check beats-ci-artifacts/pull-requests/pr-27609/elastic-agent bucket

@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 26, 2021

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Reason: Aborted from #6

  • Start Time: 2021-09-13T05:40:52.587+0000

  • Duration: 223 min 3 sec

  • Commit: b086122

Test stats 🧪

Test Results
Failed 0
Passed 53922
Skipped 5324
Total 59246

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 1

Expand to view the steps failures

filebeat-goIntegTest - mage goIntegTest
  • Took 5 min 48 sec . View more details on here
  • Description: mage goIntegTest

Log output

Expand to view the last 100 lines of log output

[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
[2021-09-13T08:02:28.961Z] 	at sun.reflect.GeneratedMethodAccessor955.invoke(Unknown Source)
[2021-09-13T08:02:28.961Z] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-09-13T08:02:28.961Z] 	at java.lang.reflect.Method.invoke(Method.java:498)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
[2021-09-13T08:02:28.961Z] 	at sun.reflect.GeneratedMethodAccessor955.invoke(Unknown Source)
[2021-09-13T08:02:28.961Z] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-09-13T08:02:28.961Z] 	at java.lang.reflect.Method.invoke(Method.java:498)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
[2021-09-13T08:02:28.961Z] 	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
[2021-09-13T08:02:28.962Z] 	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
[2021-09-13T08:02:28.962Z] 	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
[2021-09-13T08:02:28.962Z] 	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
[2021-09-13T08:02:28.962Z] 	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
[2021-09-13T08:02:28.962Z] 	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
[2021-09-13T08:02:28.962Z] 	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
[2021-09-13T08:02:28.962Z] 	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
[2021-09-13T08:02:28.962Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-09-13T08:02:28.962Z] 	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
[2021-09-13T08:02:28.962Z] 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
[2021-09-13T08:02:28.962Z] 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
[2021-09-13T08:02:28.962Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2021-09-13T08:02:28.962Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-09-13T08:02:28.962Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-09-13T08:02:28.962Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-09-13T08:02:28.962Z] 	at java.lang.Thread.run(Thread.java:748)
[2021-09-13T08:02:29.063Z] Timeout set to expire in 5 min 0 sec
[2021-09-13T08:02:29.365Z] Cleaning up /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745
[2021-09-13T08:02:29.365Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-13T08:02:29.365Z] ++ id -u
[2021-09-13T08:02:29.365Z] ++ id -g
[2021-09-13T08:02:29.365Z] + docker run -v /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-13T08:02:29.365Z] Unable to find image 'alpine:3.4' locally
[2021-09-13T08:02:29.936Z] 3.4: Pulling from library/alpine
[2021-09-13T08:02:30.196Z] c1e54eec4b57: Pulling fs layer
[2021-09-13T08:02:30.457Z] c1e54eec4b57: Download complete
[2021-09-13T08:02:30.717Z] c1e54eec4b57: Pull complete
[2021-09-13T08:02:30.717Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-09-13T08:02:30.717Z] Status: Downloaded newer image for alpine:3.4
[2021-09-13T08:02:32.630Z] + set -e
[2021-09-13T08:02:32.630Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-13T08:02:32.630Z] Change permissions with write access of all files inside the specific folder
[2021-09-13T08:02:32.630Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745
[2021-09-13T08:02:33.623Z] Running in /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745/src/github.com/elastic/beats/build
[2021-09-13T08:02:33.932Z] + rm -rf ve
[2021-09-13T08:02:33.932Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-09-13T08:02:34.260Z] + python .ci/scripts/pre_archive_test.py
[2021-09-13T08:02:44.264Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-09-13T08:02:44.265Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-09-13T08:02:44.265Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-complete/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-complete/elastic-agent-linux-amd64.docker/docker-build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-09-13T08:02:44.265Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-09-13T08:02:44.265Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-09-13T08:02:44.288Z] Running in /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745/src/github.com/elastic/beats/build
[2021-09-13T08:02:44.310Z] Recording test results
[2021-09-13T08:02:47.847Z] None of the test reports contained any result
[2021-09-13T08:02:47.856Z] [Checks API] No suitable checks publisher found.
[2021-09-13T08:02:48.220Z] + go clean -modcache
[2021-09-13T08:02:51.567Z] Timeout set to expire in 5 min 0 sec
[2021-09-13T08:02:51.874Z] Cleaning up /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745
[2021-09-13T08:02:51.874Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-13T08:02:51.874Z] ++ id -u
[2021-09-13T08:02:51.874Z] ++ id -g
[2021-09-13T08:02:51.874Z] + docker run -v /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-13T08:03:48.209Z] + set -e
[2021-09-13T08:03:48.209Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-13T08:03:48.209Z] Change permissions with write access of all files inside the specific folder
[2021-09-13T08:03:48.209Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745
[2021-09-13T08:03:48.287Z] Running in /var/lib/jenkins/workspace/PR-27609-5-2c5532cb-6364-40c0-8976-5bc31cd0e745
[2021-09-13T08:03:54.300Z] + gsutil --version
[2021-09-13T08:03:55.816Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-13T08:03:56.158Z] + gcloud auth activate-service-account --key-file ****
[2021-09-13T08:03:56.742Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-13T08:03:57.367Z] + gsutil -m -q cp eC1wYWNrL2VsYXN0aWMtYWdlbnQtcGFja2FnaW5nLWxpbnV4YjA4NjEyMjBjNTYxYmQwZmRjYzBmMDM2MDcyYzE4NDQ2Nzk2ZTk3ZQ gs://beats-ci-temp/ci/cache/
[2021-09-13T08:04:00.872Z] Scheduling project: Beats » Beats Packaging » PR-27609
[2021-09-13T08:04:04.144Z] Starting building: Beats » Beats Packaging » PR-27609 #10
[2021-09-13T09:23:53.249Z] [INFO] For detailed information see: https://beats-ci.elastic.co/job/Beats/job/packaging/job/PR-27609/10/display/redirect
[2021-09-13T09:23:54.587Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27609/src/github.com/elastic/beats
[2021-09-13T09:23:55.007Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27609
[2021-09-13T09:23:55.079Z] [INFO] getVaultSecret: Getting secrets
[2021-09-13T09:23:55.147Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-13T09:23:56.019Z] + chmod 755 generate-build-data.sh
[2021-09-13T09:23:56.019Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27609/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27609/runs/5 ABORTED 13383148
[2021-09-13T09:23:56.019Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27609/runs/5/steps/?limit=10000 -o steps-info.json
[2021-09-13T09:24:14.195Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27609/runs/5/tests/?status=FAILED -o tests-errors.json

@kvch
Copy link
Contributor

kvch commented Aug 30, 2021

What is up with this PR? Is it still required? Personally, I like this approach because it is consistent with the current packaging.

@mdelapenya
Copy link
Contributor Author

What is up with this PR? Is it still required? Personally, I like this approach because it is consistent with the current packaging.

Do you think of any incompatibility with #27621? As you suggest, I'd like to see more robustness in the build process too

@kvch
Copy link
Contributor

kvch commented Sep 1, 2021

No, I don't see any incompatibility.

* master:
  Forward port 7.14.1 changelog to master  (elastic#27687)
  Addressing multiple dashboard issues: deps loading once, field conversion, etc. (elastic#27669)
  Remove adaptive queue sizes from agent's spec files (elastic#27653)
  Osquerybeat: Improve testability and unit test coverage (elastic#27591)
  Osquerybeat: lockdown flagsfile, prevent global defaults (elastic#27611)
  Import the references of dashboard assets using the Saved Objects API (elastic#27647)
  Fix bug with override path in cgroups (elastic#27620)
  Allow Kibana client to authorize with Elasticsearch API key (elastic#27540)
  Filebeat auditd: Fix Top Exec Commands dashboard visualization (elastic#27638)
  [elastic-agent] Fix docker tar.gz generation for complete image (elastic#27621)
  Follow up changes in dashboards in mage check && fix minor issue (elastic#27553)
  [Heartbeat] Fix bug where `enabled: false` is ignored. (elastic#27615)
  Support kube_state_metrics v2.0.0 (elastic#27552)
@mdelapenya
Copy link
Contributor Author

mdelapenya commented Sep 2, 2021

jenkins run the tests please

@mdelapenya
Copy link
Contributor Author

CI is passing. @kvch @jsoriano could you please give a review on this PR?

Copy link
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please wait for one more approval.

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these changes the following images are generated, one of them containing two suffixes, and another one with a -complete suffix but seemingly having the same content as the one without this suffix. Is this expected? 🤔

REPOSITORY                                                TAG                        IMAGE ID       CREATED             SIZE
docker.elastic.co/beats/elastic-agent-complete-complete   8.0.0                      f5bc5bc3e5fd   34 seconds ago      1.55GB
docker.elastic.co/beats/elastic-agent                     8.0.0                      4069689b3330   3 minutes ago       523MB
docker.elastic.co/beats/elastic-agent-complete            8.0.0                      61d02e44734e   3 minutes ago       523MB
docker.elastic.co/beats/elastic-agent-ubi8                8.0.0                      51cecc7df1fb   3 minutes ago       318MB

@mdelapenya
Copy link
Contributor Author

With these changes the following images are generated, one of them containing two suffixes, and another one with a -complete suffix but seemingly having the same content as the one without this suffix. Is this expected? 🤔

I think because #27621 was merged before this one, there are conflicts in resolving the issue described here. As @kvch said, these changes are affecting configuration, while that PR is updating the build system to cover this use case.

I tested my changes before #27621, I can give it a try again, with and without that commit.

* master: (39 commits)
  [Heartbeat] Move JSON tests from python->go (elastic#27816)
  docs: simplify permissions for Dockerfile COPY (elastic#27754)
  Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789)
  [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774)
  warn log entry and no validation failure when both queue_url and buck… (elastic#27612)
  libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178)
  [Heartbeat] Add httpcommon options to ZipURL (elastic#27699)
  Add a header round tripper option to httpcommon (elastic#27509)
  [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779)
  Rename dashboards according to module.yml files for master (elastic#27749)
  Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726)
  Accept syslog dates with leading 0 (elastic#27775)
  [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727)
  [Filebeat] Threatintel compatibility updates (elastic#27323)
  Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707)
  [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491)
  Fix flakyness of TestFilestreamEmptyLine (elastic#27705)
  [Filebeat] kafka v2 using parsers (elastic#27335)
  Update Kafka version parsing / supported range (elastic#27720)
  Update Sarama to 1.29.1 (elastic#27717)
  ...
* master:
  [Auditbeat] scanner honor include_files (elastic#27722)
  chore(ci): remove not used param when triggering e2e tests (elastic#27823)
  LineReader: Reuse temporary buffer to reduce per-line allocation (elastic#27782)
@mdelapenya
Copy link
Contributor Author

@jsoriano @kvch @andrewkroh @andrewvc I updated this PR and verified that:

cd x-pack/elastic-agent
PLATFORMS="linux/amd64" mage package
  • the Docker engine contains 3 Docker images, default, complete and ubi8, each of them with the right size (complete is around 1'5Gb)
  • generates the Docker binaries for each variant: the build/distributions directory contains 3 tar.gz files, one per variant (default, complete and ubi8), each of them with the right size (complete includes Node libs)
$ ls -l x-pack/elastic-agent/build/distributions/
total 2791688
-rw-r--r--@ 1 mdelapenya  staff  119939138  9 Sep 17:18 elastic-agent-8.0.0-amd64.deb
-rw-r--r--  1 mdelapenya  staff        159  9 Sep 17:18 elastic-agent-8.0.0-amd64.deb.sha512
-rw-r--r--  1 mdelapenya  staff  244353172  9 Sep 17:18 elastic-agent-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        175  9 Sep 17:18 elastic-agent-8.0.0-linux-amd64.docker.tar.gz.sha512
-rw-r--r--  1 mdelapenya  staff  120003980  9 Sep 17:17 elastic-agent-8.0.0-linux-x86_64.tar.gz
-rw-r--r--  1 mdelapenya  staff        169  9 Sep 17:17 elastic-agent-8.0.0-linux-x86_64.tar.gz.sha512
-rw-r--r--  1 mdelapenya  staff  119745407  9 Sep 17:18 elastic-agent-8.0.0-x86_64.rpm
-rw-r--r--  1 mdelapenya  staff        160  9 Sep 17:18 elastic-agent-8.0.0-x86_64.rpm.sha512
-rw-r--r--  1 mdelapenya  staff  615426504  9 Sep 17:23 elastic-agent-complete-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        184  9 Sep 17:24 elastic-agent-complete-8.0.0-linux-amd64.docker.tar.gz.sha512
-rw-r--r--  1 mdelapenya  staff  163243411  9 Sep 17:18 elastic-agent-ubi8-8.0.0-linux-amd64.docker.tar.gz
-rw-r--r--  1 mdelapenya  staff        180  9 Sep 17:18 elastic-agent-ubi8-8.0.0-linux-amd64.docker.tar.gz.sha512
  • Docker loading each of these artifact results in having the Docker engine loading the right image:
$ docker images | grep agent | grep -v SNAP
docker.elastic.co/beats/elastic-agent-complete          8.0.0                                      7cc90f716192   40 minutes ago      1.55GB
docker.elastic.co/beats/elastic-agent-ubi8              8.0.0                                      58ed21edadcd   44 minutes ago      297MB
docker.elastic.co/beats/elastic-agent                   8.0.0                                      a9c9517dcace   44 minutes ago      521MB

TBH, I'm pretty sure I'm acting as a surgeon in their first to-open-heart surgery with this PR affecting the build system. I'd appreciate your review. Thanks!

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I like that variants are more just another image in the yml file and less a special case in dockerbuilder.go.

Added some comments about removing unneeded code and fully reverting #27621. I wonder if we can even remove more code for variants in the docker builder.

x-pack/osquerybeat/beater/config_plugin_test.go Outdated Show resolved Hide resolved
@@ -48,7 +48,7 @@ const (
packageStagingDir = "build/package"

// defaultBinaryName specifies the output file for zip and tar.gz.
defaultBinaryName = "{{.Name}}-{{if .Variant}}{{.Variant}}-{{end}}{{.Version}}{{if .Snapshot}}-SNAPSHOT{{end}}{{if .OS}}-{{.OS}}{{end}}{{if .Arch}}-{{.Arch}}{{end}}"
defaultBinaryName = "{{.Name}}-{{.Version}}{{if .Snapshot}}-SNAPSHOT{{end}}{{if .OS}}-{{.OS}}{{end}}{{if .Arch}}-{{.Arch}}{{end}}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Variant is not needed anymore here, I guess that the rest of #27621 can be reverted too?

dev-tools/mage/dockerbuilder.go Outdated Show resolved Hide resolved
@mergify
Copy link
Contributor

mergify bot commented Sep 22, 2021

This pull request does not have a backport label. Could you fix it @mdelapenya? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021
@mdelapenya
Copy link
Contributor Author

Once #28134 is merged, we should update this PR and possibly resolve conflicts

@mergify
Copy link
Contributor

mergify bot commented Oct 6, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 27608-fix-elastic-agent-build upstream/27608-fix-elastic-agent-build
git merge upstream/master
git push upstream 27608-fix-elastic-agent-build

@mdelapenya
Copy link
Contributor Author

Overridden by #28526

@mdelapenya mdelapenya closed this Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify bug build-system Issue or change affecting Mage, Make, or build scripts. Team:Automation Label for the Observability productivity team Team:Elastic-Agent Label for the Agent team Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Complete variant for the elastic agent is not generating the right binaries
6 participants