Signing using Infra pipeline #621
Conversation
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
.ci/Jenkinsfile
Outdated
| pattern: 'build/integrations/*.zip', | ||
| sharedPublicly: false, | ||
| showInline: true) | ||
| build(wait: true, propagate: true, job: 'elastic+unified-release+master+sign-artifacts-with-gpg', |
There was a problem hiding this comment.
Hey @v1v,
I think I have just understood the architecture used by the apm-agent-php to sign their artifacts. It's a bit different than I imagined :) Let me know if I'm correct - the signing procedure is executed in the internal-ci, but not only the signing step but the entire pipeline (see: internal-ci, apm-ci).
Let's consider the elastic/integrations:
I was wondering if there is an option to support it without re-running the entire Integrations pipeline in the internal-ci, but just call the signing pipeline there?
Otherwise, I understand that we need a similar JJB in the internal-ci for elastic/integrations, which will run just building (not testing), signing, and publishing to the Package Storage.
There was a problem hiding this comment.
This means that I can close this PR - I tried to build a PoC or something similar.
There was a problem hiding this comment.
Another thought:
is there an option to use the remote API? so that any Elastic pipeline (with auth) can trigger a build in the internal-ci instance?
There was a problem hiding this comment.
is there an option to use the remote API? so that any Elastic pipeline (with auth) can trigger a build in the internal-ci instance?
IIRC, I think there is no such a integration, though let me dig a bit into what we discussed for the php agent in case those questions were also answered.
There was a problem hiding this comment.
Thanks! I'm afraid that it will force package owners to create artificial pipelines just to reach out to the internal-ci pipeline, which seems to be an unnecessary step if we have such an API.
There was a problem hiding this comment.
I'm afraid I could not find something that validates my hypothesis , so I just asked the infra-automation team, you are also poked in that particular conversation. To be clear, the conversation is outside of this thread, but I'll comment here the outcome
|
/test |
| useCrumbCache: true, | ||
| useJobInfoCache: true) | ||
| } | ||
| googleStorageDownload(bucketUri: "${env.INFRA_SIGNING_BUCKET_SIGNED_ARTIFACTS_PATH}/*", |
There was a problem hiding this comment.
Is this downloading all the packages ever signed? Or there is some cleanup process for the signed artifacts path?
There was a problem hiding this comment.
To be ownest with you I have never tried downloading everything in that bucket as it could kill the CI worker :D It's the same job we're using to sign Beats artifacts.
| triggerRemoteJob(auth: CredentialsAuth(credentials: 'local-readonly-api-token'), | ||
| job: 'https://internal-ci.elastic.co/job/elastic+unified-release+master+sign-artifacts-with-gpg', | ||
| token: TOKEN, | ||
| parameters: "gcs_input_path=${env.INFRA_SIGNING_BUCKET_ARTIFACTS_PATH}", |
There was a problem hiding this comment.
Out of curiosity, does this job check that this path belongs to elastic? In case somehow a malicious pipeline triggers this job to sign anything placed in a bucket with public access.
There was a problem hiding this comment.
The job is configured in the internal-ci instance, so the malicious user should be also internal?
There was a problem hiding this comment.
I was wondering more about a change in the Jenkinsfile in a PR.
| credentialsId: env.INTERNAL_CI_JOB_GCS_CREDENTIALS, | ||
| localDirectory: "${env.INTEGRATIONS_SIGNATURES_PATH}/", | ||
| pathPrefix: "${env.INFRA_SIGNING_BUCKET_SIGNED_ARTIFACTS_SUBFOLDER}") | ||
| sh(label: 'Rename .asc to .sig', script: 'for f in ' + "${env.INTEGRATIONS_SIGNATURES_PATH}" + '/*.asc; do mv "$f" "${f%.asc}.sig"; done') |
There was a problem hiding this comment.
It might be nice to encapsulate these steps of uploading, signing, downloading and renaming into a single signArtifacts helper 🙂
There was a problem hiding this comment.
Yes, they will be encapsulated. I'm planning to wrap (signing + push to Package Storage) into a single command. I have to make sure the push works correctly.
4 participants
This PR modifies the Jenkinsfile to call the Infra singing pipeline to sign package artifacts using Elastic public key.
Summing up:
CI test builds the package, cross-call the internal CI to sign it, fetches signature, and archives them. It proves that we can freely call the internal-ci pipeline from beats-ci.