Allow users to specify different credentials for source and target

[iansrobinson]

neptune-export gets the credentials for accessing the Neptune Management AIP using the default provider chain (using env vars, system properties, Web identity token provider, profile, and EC2 container credentials, in that order). When querying an IAM auth enabled database, it looks for the necessary access key and secret key in the env vars and then system properties.

However, there are situations in which neptune-export may need a separate set of credentials to access the target (Amazon S3, or an Amazon Kinesis Data Stream). This can happen when the tool is reading data from Neptune in one account and writing to a target in another account.

This feature would allows users to specify different credentials for source and target.

[Cole-Greer]

I would like to propose that we support the ability to assume a separate role uploading to the target instead of using 2 separate sets of credentials. This would be more aligned with best practices than passing in 2 separate sets of credentials. My suggestion is to add new configuration options --target-assume-role-arn and -target-assume-role-external-id where users can specify a role ARN to use and optionally an external id as well. Neptune Export would then attempt to assume this role via the credentials found using the default provider chain. This would allow a separate role to be used for the main export and the target upload.

@alxcnt As this issue originated from your request, do you have any thoughts on this approach?

[alxcnt]

Hello @Cole-Greer thanks for looking into this.
Here were my thoughts on this: awslabs/amazon-neptune-tools#307 (comment) the idea was that as we can specify everything via env vars already and this case is relatively rare(not sure) we could just specify the prefix for the alternative var set so one could use the same resolution mechanism that already exists if the target account is different.

[Cole-Greer]

Thanks @alxcnt,
The reason I'm suggesting a role-assuming based solution instead of a prefix for environment variable credentials is that it is more aligned with AWS best practices and in my opinion offers better usability. The security case for using cross-account roles is well highlighted in this blog post. Adding the ability to assume a separate role for target uploads will enable the same cross account access use case without the risks associated with managing a second set of credentials.

I also see from issue #16 that your setup already involves assuming roles. The credentials generated through role assumption are temporary and by default timeout after 1 hour. For large exports this is not enough time and credentials will need to be refreshed. Credential refreshing can be done if role assumption takes place within Neptune Export but not if credentials are only passed in externally.

To extend my previous proposal, we could also pass in the role to use for target uploads via a TARGET_AWS_ROLE_ARN environment variable. To use this setup, you would need a role in your target account with the necessary permissions (in this case permission to write to Kinesis). That role would need to have a trust policy granting access to your user in your main account. Your main account user would also need permisisons to assume that role. I would be happy to provide some examples of how to do this if you would like. From then you could run Neptune Export as always (main account user credentials) while setting TARGET_AWS_ROLE_ARN in order to upload to a Kinesis stream in a separate account. Please let me know if there are any reasons this approach would not fulfill your needs and we will adjust this plan accordingly.

[alxcnt]

Hello @Cole-Greer the thing with the role-based auth is that sometimes you'd like to use access_key + secret_key and don't want to set up separate roles etc, so perhaps it needs to be taken into account. You approach will definitely work.

[Cole-Greer]

Hi @alxcnt, I understand that role-based auth does require slightly more initial setup but I believe it is justified by the security benefits highlighted above. We will proceed with adding support for role-based auth for exporting to both Kinesis and S3.

[Cole-Greer]

Resolved with #35, users can now assume a different role for target uploads to S3 or Kinesis. (uses new --stream-role-arn or S3RoleArn parameters)