Skip to content

[FLINK-39865][ci] Replace abandoned travis-ci artifacts uploader with AWS CLI for nightly S3 upload#28327

Merged
MartijnVisser merged 1 commit into
apache:masterfrom
MartijnVisser:FLINK-39865
Jun 5, 2026
Merged

[FLINK-39865][ci] Replace abandoned travis-ci artifacts uploader with AWS CLI for nightly S3 upload#28327
MartijnVisser merged 1 commit into
apache:masterfrom
MartijnVisser:FLINK-39865

Conversation

@MartijnVisser

@MartijnVisser MartijnVisser commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

The nightly Upload artifacts to S3 step (flink-ci.flink-master-mirror, definitionId=1) has failed on every run since 2026-06-02:

/home/vsts_azpcontainer/bin/artifacts: line 1: syntax error near unexpected token `newline'
/home/vsts_azpcontainer/bin/artifacts: line 1: `<?xml version="1.0" encoding="UTF-8"?>'
##[error]Bash exited with code '2'.

tools/ci/deploy_nightly_to_s3.sh installs the travis-ci/artifacts uploader via curl ... | bash. That installer downloads its binary from a hardcoded travis-ci S3 bucket (s3.amazonaws.com/travis-ci-gmbh/artifacts/...) which now returns HTTP 403 AccessDenied. Because the install writes the response without checking it, the XML error page is saved as the artifacts binary and executed, producing the syntax error above. The travis-ci/artifacts project is abandoned.

Failure history (nightly cron runs):

Build Date Result
75662 2026-06-05 failed
75618 2026-06-04 failed
75582 2026-06-03 failed
75557 2026-06-02 failed
75530 2026-06-01 succeeded

Brief change log

  • Replace the travis-ci/artifacts uploader in tools/ci/deploy_nightly_to_s3.sh with aws s3 cp "$FILES_DIR" "s3://$ARTIFACTS_S3_BUCKET/" --recursive. The recursive copy to the bucket root preserves the previous --target-paths / layout, and the bucket's default (private) ACL matches the travis uploader's default.
  • Install AWS CLI v2 via the official installer when aws is not already present in the build container (chesnay/flink-ci does not ship it, but provides curl and unzip).
  • In tools/azure-pipelines/build-nightly-dist.yml, map the existing Azure secret variables to AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, which the AWS CLI reads from the environment. Credentials are no longer passed as command-line arguments, so they no longer appear in the set -x trace.

Verifying this change

This change is a CI/build-tooling fix and is verified by the nightly pipeline itself. Additionally validated locally:

  • bash -n tools/ci/deploy_nightly_to_s3.sh passes.
  • A dry run (aws s3 cp <dir> s3://<bucket>/ --recursive --dryrun) confirms the recursive upload preserves the relative path layout (files land at the bucket root, sub-directories preserved).
  • Confirmed via a set -x trace that the credential values do not appear on the aws s3 cp command line.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no (removes an abandoned external download; uses AWS CLI already expected in the deploy environment)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code, Opus 4.8)

Generated-by: Claude Code (Opus 4.8)

@flinkbot

flinkbot commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

… AWS CLI for nightly S3 upload

The nightly "Upload artifacts to S3" step installed the travis-ci/artifacts
uploader via `curl ... | bash`. That installer downloads its binary from a
hardcoded travis-ci S3 bucket which now returns HTTP 403, so the XML error
page was saved as the `artifacts` binary and executed, breaking every nightly
run since 2026-06-02. The travis-ci/artifacts project is abandoned.

Replace it with the maintained AWS CLI (`aws s3 cp --recursive`), installing
AWS CLI v2 in the build container when absent. Credentials are now read from
the environment (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) instead of being
passed as command-line arguments, so they no longer appear in the `set -x`
trace. The default (private) ACL and bucket-root upload layout are preserved.

Generated-by: Claude Code (Opus 4.8)

@snuyanzin snuyanzin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
thanks for fixing this

@MartijnVisser MartijnVisser merged commit 5c0e9a8 into apache:master Jun 5, 2026
@MartijnVisser MartijnVisser deleted the FLINK-39865 branch June 5, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants