Skip to content

[WIP] Add localstack test to S3 package#587

Closed
grouville wants to merge 1 commit into
dagger:mainfrom
grouville:localstackS3
Closed

[WIP] Add localstack test to S3 package#587
grouville wants to merge 1 commit into
dagger:mainfrom
grouville:localstackS3

Conversation

@grouville
Copy link
Copy Markdown
Member

Localstack test implementation.

As I didn't rebase frequently, I need to add the S3#Sync tonight, but currently stuck by the issues cited below.

I am encountering these 2 issues :

  • A race condition (Through my tests/debugging, I realized that we cp / sync a file/ folder to AWS S3, then we query it right away with a recursive ls. However, it triggered an error (not very clear) that the file doesn't exist (fixed with a sleep, but if behavior is confirmed, will need to find a better fix). According to me, it was hidden by the fact that we used to always query pre-existing files.
  • The tests totally work on my personal AWS accounts, but when using CI creds, it seems to break. Seen with Sam, but I don't know how to fix it.

Prerequisites :
For the LocalStack test to run, you need to have LocalStack running on port 4566 : pip install localstack, then localstack start. Once running, you also need to create the dagger-ci bucket : aws --endpoint-url=http://localhost:4566 s3 mb s3://dagger-ci.

@grouville grouville requested a review from samalba June 8, 2021 10:53
@grouville grouville self-assigned this Jun 8, 2021
…with CI creds

Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>
@grouville
Copy link
Copy Markdown
Member Author

grouville commented Jun 8, 2021

@samalba, on the first run. The CI broke, but not related to AWS tests (Kubernetes and Docker Run) : ultra weird that it doesn't work locally using CI creds. If it works again, I'll wipe everything and start again with a clean setup

@grouville grouville changed the title WIP: Implement localstack to S3 package. Fully fonctional on my side [WIP] Implement localstack to S3 package. Fully fonctional on my side Jun 8, 2021
@grouville grouville changed the title [WIP] Implement localstack to S3 package. Fully fonctional on my side [WIP] Add localstack test to S3 package. (Fully fonctional on my side) Jun 8, 2021
@grouville grouville changed the title [WIP] Add localstack test to S3 package. (Fully fonctional on my side) [WIP] Add localstack test to S3 package Jun 8, 2021
@aluzzardi
Copy link
Copy Markdown
Contributor

@samalba, on the first run. The CI broke, but not related to AWS tests (Kubernetes and Docker Run) : ultra weird that it doesn't work locally using CI creds. If it works again, I'll wipe everything and start again with a clean setup

Yep, we've had CI issues, should be fixed now by rebasing.

NOTE: you'll need to remove random.cue if you created one and use dagger.io/random instead (just like in Kubernetes)

@grouville grouville mentioned this pull request Aug 19, 2021
@grouville grouville closed this Aug 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants