-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
25 changed files
with
651 additions
and
401 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,130 @@ | ||
# S3 Support | ||
|
||
When installed with the S3 addon `spock` will attempt to identify S3 URI(s) (e.g. `s3://<bucket-name>/<key>`) and handle | ||
them automatically. The user only needs to provide an active `boto3.session.Session` to an `S3Config` object and pass | ||
it to the `ConfigArgBuilder`. | ||
|
||
|
||
### Installing | ||
|
||
Install `spock` with the extra s3 related dependencies. | ||
|
||
```bash | ||
pip install spock-config[s3] | ||
``` | ||
|
||
### Creating a boto3 Session | ||
|
||
The user must provide an active `boto3.session.Session` object to `spock` in order for the library to automatically | ||
handle S3 URI(s). Configuration is **highly dependent** upon your current AWS setup/security. Please refer to the | ||
`boto3` docs for [session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) and | ||
[credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for help on how to | ||
correctly configure your `boto3.session.Session`. | ||
|
||
For instance, let's just suppose we are going to get our tokens via SAML authorization | ||
where we already have the SAMLAssertion, RoleArn, and PrincipalArn stored as env variables: | ||
|
||
```python | ||
import boto3 | ||
import os | ||
|
||
client = boto3.client('sts') | ||
|
||
token = client.assume_role_with_saml( | ||
RoleArn=os.environ.get("RoleArn"), PrincipalArn=os.environ.get("PrincipalArn"), | ||
SAMLAssertion=os.environ.get("SamlString") | ||
) | ||
|
||
credentials = token['Credentials'] | ||
|
||
session = boto3.Session( | ||
aws_access_key_id=credentials['AccessKeyId'], | ||
aws_secret_access_key=credentials['SecretAccessKey'], | ||
aws_session_token=credentials['SessionToken'], | ||
region_name=os.environ.get('AWS_REGION')) | ||
``` | ||
|
||
### Using the S3Config Object | ||
|
||
As an example let's create a basic `@spock` decorated class, instantiate a `S3Config` object from `spock.addons` with | ||
the `boto3.session.Session` we created above, and pass it to the `ConfigArgBuilder`. | ||
|
||
```python | ||
from spock.addons import S3Config | ||
from spock.builder import ConfigArgBuilder | ||
from spock.config import spock | ||
from typing import List | ||
|
||
@spock | ||
class BasicConfig: | ||
"""Basic spock configuration for example purposes | ||
Attributes: | ||
parameter: simple boolean that flags rounding | ||
fancy_parameter: parameter that multiplies a value | ||
fancier_parameter: parameter that gets added to product of val and fancy_parameter | ||
most_fancy_parameter: values to apply basic algebra to | ||
""" | ||
parameter: bool | ||
fancy_parameter: float | ||
fancier_parameter: float | ||
most_fancy_parameter: List[int] | ||
|
||
def main(): | ||
# Create an S3Config object and pass in the boto3 session | ||
s3_config = S3Config( | ||
session=session | ||
) | ||
# Chain the generate function to the ConfigArgBuilder call | ||
# Pass in the S3Config object | ||
config = ConfigArgBuilder( | ||
BasicConfig, | ||
desc='S3 example', | ||
s3_config=s3_config | ||
).generate() | ||
``` | ||
|
||
### Defining the configuration file with a S3 URI | ||
|
||
Usually we pass a relative or absolute system path as the configuration file command line argument. Here we pass | ||
in a S3 URI instead: | ||
|
||
```bash | ||
$ python simple.py -c s3://my-bucket/path/to/file/config.yaml | ||
``` | ||
|
||
With a `S3Config` object passed into the `ConfigArgBuilder` the S3 URI will automatically be handled by `spock`. | ||
|
||
### Saving to a S3 URI | ||
|
||
Similarly, we usually pass a relative or absolute system path to the `SavePath` special argument type or | ||
to the `user_specified_path` kwarg. Again, instead we give a S3 URI: | ||
|
||
```python | ||
def main(): | ||
# Create an S3Config object and pass in the boto3 session | ||
s3_config = S3Config( | ||
session=session | ||
) | ||
# Chain the generate function to the ConfigArgBuilder call | ||
# Pass in the S3Config object | ||
config = ConfigArgBuilder( | ||
BasicConfig, | ||
desc='S3 example', | ||
s3_config=s3_config | ||
).save(user_specified_path="s3://my-bucket/path/to/file/").generate() | ||
``` | ||
|
||
With a `S3Config` object passed into the `ConfigArgBuilder` the S3 URI will automatically be handled by `spock`. | ||
|
||
### S3Transfer ExtraArgs | ||
|
||
If you require any other settings for uploading or downloading files from S3 the `S3Config` class has two extra | ||
attributes: | ||
|
||
`download_config` which takes a `S3DownloadConfig` object from `spock.addons` which supports all ExtraArgs from | ||
[S3Transfer.ALLOWED_DOWNLOAD_ARGS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_DOWNLOAD_ARGS) | ||
|
||
`upload_config` which takes a `S3UploadConfig` object from `spock.addons` which supports all ExtraArgs from | ||
[S3Transfer.ALLOWED_UPLOAD_ARGS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# -*- coding: utf-8 -*- | ||
|
||
# Copyright 2019 FMR LLC <opensource@fidelity.com> | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
""" | ||
Spock is a framework that helps manage complex parameter configurations for Python applications | ||
Please refer to the documentation provided in the README.md | ||
""" | ||
from spock.addons.s3.utils import S3Config | ||
from spock.addons.s3.configs import S3DownloadConfig | ||
from spock.addons.s3.configs import S3UploadConfig | ||
|
||
__all__ = ["s3", "S3Config", "S3DownloadConfig", "S3UploadConfig"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# -*- coding: utf-8 -*- | ||
|
||
# Copyright 2019 FMR LLC <opensource@fidelity.com> | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
""" | ||
Spock is a framework that helps manage complex parameter configurations for Python applications | ||
Please refer to the documentation provided in the README.md | ||
""" | ||
|
||
__all__ = ["utils"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
# -*- coding: utf-8 -*- | ||
|
||
# Copyright 2019 FMR LLC <opensource@fidelity.com> | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
"""Handles all S3 related configurations""" | ||
|
||
import attr | ||
try: | ||
import boto3 | ||
from botocore.client import BaseClient | ||
from s3transfer.manager import TransferManager | ||
except ImportError: | ||
print('Missing libraries to support S3 functionality. Please re-install spock with the extra s3 dependencies -- ' | ||
'pip install spock-config[s3]') | ||
import typing | ||
|
||
|
||
# Iterate through the allowed download args for S3 and map into optional attr.ib | ||
download_attrs = { | ||
val: attr.ib( | ||
default=None, | ||
type=str, | ||
validator=attr.validators.optional(attr.validators.instance_of(str)) | ||
) for val in TransferManager.ALLOWED_DOWNLOAD_ARGS} | ||
|
||
|
||
# Make the class dynamically | ||
S3DownloadConfig = attr.make_class(name="S3DownloadConfig", attrs=download_attrs, kw_only=True, frozen=True) | ||
|
||
# Iterate through the allowed upload args for S3 and map into optional attr.ib | ||
upload_attrs = { | ||
val: attr.ib( | ||
default=None, | ||
type=str, | ||
validator=attr.validators.optional(attr.validators.instance_of(str)) | ||
) for val in TransferManager.ALLOWED_UPLOAD_ARGS | ||
} | ||
|
||
|
||
# Make the class dynamically | ||
S3UploadConfig = attr.make_class(name="S3UploadConfig", attrs=upload_attrs, kw_only=True, frozen=True) | ||
|
||
|
||
@attr.s(auto_attribs=True) | ||
class S3Config: | ||
"""Configuration class for S3 support | ||
*Attributes*: | ||
session: instantiated boto3 session object | ||
s3_session: automatically generated s3 client from the boto3 session | ||
kms_arn: AWS KMS key ARN (optional) | ||
temp_folder: temporary working folder to write/read spock configuration(s) (optional: defaults to /tmp) | ||
""" | ||
session: boto3.Session | ||
s3_session: BaseClient = attr.ib(init=False) | ||
temp_folder: typing.Optional[str] = '/tmp/' | ||
download_config: S3DownloadConfig = S3DownloadConfig() | ||
upload_config: S3UploadConfig = S3UploadConfig() | ||
|
||
def __attrs_post_init__(self): | ||
self.s3_session = self.session.client('s3') |
Oops, something went wrong.