Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DEVOPS-836] configure hydra to upload everything to S3 #352

Merged
merged 1 commit into from Jun 27, 2018

Conversation

Projects
None yet
2 participants
@cleverca22
Copy link
Contributor

commented Jun 6, 2018

this PR configures hydra to upload all artifacts to the S3 bucket iohk-nix-cache
it relies on an IAM user that has S3 access to the bucket
and the access keyid/secret have to be configured in /var/lib/hydra/queue-runner/.aws/credentials
that can be automated with the following commands

sudo -u hydra-queue-runner -i
nix run nixpkgs.awscli
aws configure

the PR also handles redirections of binary-cache files from hydra.iohk.io to https://iohk-nix-cache.s3-eu-central-1.amazonaws.com

@cleverca22 cleverca22 added the WIP label Jun 6, 2018

@cleverca22 cleverca22 force-pushed the hydra-s3 branch from 40774e8 to dc2980b Jun 6, 2018

@cleverca22 cleverca22 force-pushed the hydra-s3 branch from dc2980b to 107ef78 Jun 6, 2018

@cleverca22 cleverca22 changed the title [DEVOPS-836] test out putting the binary cache on S3 [DEVOPS-836] configure hydra to upload everything to S3 Jun 6, 2018

@cleverca22

This comment has been minimized.

Copy link
Contributor Author

commented Jun 8, 2018

the CORS policy on the S3 bucket also has to be updated like so:

<CORSConfiguration>
	<CORSRule>
		<AllowedOrigin>https://hydra.iohk.io</AllowedOrigin>
		<AllowedMethod>GET</AllowedMethod>
		<MaxAgeSeconds>3000</MaxAgeSeconds>
		<AllowedHeader>Authorization</AllowedHeader>
	</CORSRule>
</CORSConfiguration>

without that, the JS in the hydra ui is unable to load derivation logs from S3

@cleverca22 cleverca22 merged commit 1c845bb into master Jun 27, 2018

35 checks passed

buildkite/iohk-ops Build #890 passed (7 minutes, 37 seconds)
Details
buildkite/iohk-ops/iohk-ops Passed (7 minutes, 25 seconds)
Details
buildkite/iohk-ops/iohk-ops-shellcheck Passed (4 seconds)
Details
buildkite/iohk-ops/pipeline Passed (5 seconds)
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-report-server-static.x86_64-linux Hydra build #156537 of serokell:iohk-nixops-352:cardano-report-server-static.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-auxx.x86_64-darwin Hydra build #156538 of serokell:iohk-nixops-352:cardano-sl-auxx.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-auxx.x86_64-linux Hydra build #156558 of serokell:iohk-nixops-352:cardano-sl-auxx.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-explorer-static.x86_64-linux Hydra build #156544 of serokell:iohk-nixops-352:cardano-sl-explorer-static.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-node-static.x86_64-darwin Hydra build #156553 of serokell:iohk-nixops-352:cardano-sl-node-static.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-node-static.x86_64-linux Hydra build #156548 of serokell:iohk-nixops-352:cardano-sl-node-static.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-tools.x86_64-darwin Hydra build #156549 of serokell:iohk-nixops-352:cardano-sl-tools.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-tools.x86_64-linux Hydra build #156550 of serokell:iohk-nixops-352:cardano-sl-tools.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-wallet-new.x86_64-darwin Hydra build #156547 of serokell:iohk-nixops-352:cardano-sl-wallet-new.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-wallet-new.x86_64-linux Hydra build #156568 of serokell:iohk-nixops-352:cardano-sl-wallet-new.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl-wallet.x86_64-linux Hydra build #156566 of serokell:iohk-nixops-352:cardano-sl-wallet.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:cardano-sl.x86_64-linux Hydra build #156543 of serokell:iohk-nixops-352:cardano-sl.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.mainnetExplorer.x86_64-darwin Hydra build #156551 of serokell:iohk-nixops-352:connectScripts.mainnetExplorer.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.mainnetExplorer.x86_64-linux Hydra build #156546 of serokell:iohk-nixops-352:connectScripts.mainnetExplorer.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.mainnetWallet.x86_64-darwin Hydra build #156556 of serokell:iohk-nixops-352:connectScripts.mainnetWallet.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.mainnetWallet.x86_64-linux Hydra build #156536 of serokell:iohk-nixops-352:connectScripts.mainnetWallet.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.stagingExplorer.x86_64-darwin Hydra build #156552 of serokell:iohk-nixops-352:connectScripts.stagingExplorer.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.stagingExplorer.x86_64-linux Hydra build #156562 of serokell:iohk-nixops-352:connectScripts.stagingExplorer.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.stagingWallet.x86_64-darwin Hydra build #156541 of serokell:iohk-nixops-352:connectScripts.stagingWallet.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:connectScripts.stagingWallet.x86_64-linux Hydra build #156542 of serokell:iohk-nixops-352:connectScripts.stagingWallet.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:daedalus-bridge.x86_64-darwin Hydra build #156560 of serokell:iohk-nixops-352:daedalus-bridge.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:daedalus-bridge.x86_64-linux Hydra build #156540 of serokell:iohk-nixops-352:daedalus-bridge.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:iohk-ops.x86_64-linux Hydra build #156565 of serokell:iohk-nixops-352:iohk-ops.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:nixops.x86_64-linux Hydra build #156554 of serokell:iohk-nixops-352:nixops.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:nixosTests.walletAPI Hydra build #156557 of serokell:iohk-nixops-352:nixosTests.walletAPI
Details
continuous-integration/hydra:serokell:iohk-nixops-352:nixpkgs Hydra build #156545 of serokell:iohk-nixops-352:nixpkgs
Details
continuous-integration/hydra:serokell:iohk-nixops-352:purescript.x86_64-darwin Hydra build #156555 of serokell:iohk-nixops-352:purescript.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:purescript.x86_64-linux Hydra build #156567 of serokell:iohk-nixops-352:purescript.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:stack2nix.x86_64-darwin Hydra build #156559 of serokell:iohk-nixops-352:stack2nix.x86_64-darwin
Details
continuous-integration/hydra:serokell:iohk-nixops-352:stack2nix.x86_64-linux Hydra build #156539 of serokell:iohk-nixops-352:stack2nix.x86_64-linux
Details
continuous-integration/hydra:serokell:iohk-nixops-352:stagingWalletdockerImage Hydra build #156563 of serokell:iohk-nixops-352:stagingWalletdockerImage
Details

@cleverca22 cleverca22 deleted the hydra-s3 branch Jun 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.