Integrate S3Interface with replicator #72
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sarahwooders
changed the title
parasj
reviewed
Jan 17, 2022
| @@ -223,9 +226,22 @@ def start_gateway( | |||
| docker_out, docker_err = self.run_command(f"sudo docker pull {gateway_docker_image}") | |||
| assert "Status: Downloaded newer image" in docker_out or "Status: Image is up to date" in docker_out, (docker_out, docker_err) | |||
|
|
|||
| # read AWS config file to get credentials | |||
| # TODO: Integrate this with updated skylark config file | |||
There was a problem hiding this comment.
One way to easily integrate configs would be to pass all of the config variables here from config.json. Feel free to open an issue linking this area of the code so we don't block this PR.
There was a problem hiding this comment.
Made an issue #85
| credential_provider = AwsCredentialsProvider.new_default_chain(bootstrap) | ||
|
|
||
| # get aws auth info for docker envs | ||
| aws_access_key_id = os.getenv("AWS_ACCESS_KEY_ID", None) |
There was a problem hiding this comment.
You can use something like this to add a way to load credentials from the config.json as well:
from skylark.cli.cli_helper import load_config
config = load_config()
return config.get("aws_access_key_id"), config.get("aws_secret_access_key")
|
It seems to be working now (see my last commit for some minor changes): |
parasj
approved these changes
Jan 17, 2022
parasj
reviewed
Jan 18, 2022
parasj
reviewed
Jan 18, 2022
|
I've started getting an authentication error with the VPCs, which I'm guessing might be happening from too many boto3 requests (since the commands work fine in the CLI). I stopped the errors by commenting out all but the US regions, but that's probably not the fix we want... |
* Reuse S3Interface, batch S3 delete calls to fix AWS bug * When getting object sizes, throttle requests to avoid retry errors * Types in Chunk should be Optional Co-authored-by: Sarah Wooders <sarahwooders@gmail.com>
|
Are we good to merge this PR? I think we can just make the config stuff another GitHub issue and table it for now. |
sarahwooders
added a commit
that referenced
this pull request
Jan 20, 2022
This reverts commit f3cf3df.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
src_object_storeanddst_object_storeas additional options for chunk location.S3Interfacefor object store hopsS3Interface(using the originalAwsCredentialsProvider.new_default_chainwith credentials copied to the gateway didn't work for unknown reasons)Potential Issues
GatewaySenderandGatewayReceivern_chunks(only tested with--n-chunks 100) because ofawscrterror