Polystores is an abstraction and a collection of clients to interact with cloud storages.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
polystores Fix import order Feb 18, 2019
requirements Bump version to 0.0.4 Jan 10, 2019
tests Update deletion process Feb 11, 2019
.gitignore Bootstrap project Dec 10, 2018
.prospector.yaml Bootstrap project Dec 10, 2018
.pyup.yml
.travis.yml Bootstrap project Dec 10, 2018
CONTRIBUTING.md
LICENSE Bootstrap project Dec 10, 2018
MANIFEST.in Bump version to 0.0.3 Jan 10, 2019
README.md
pytest.ini Bootstrap project Dec 10, 2018
setup.cfg Fix imports Jan 23, 2019
setup.py Bump version to 0.1.5 (re-raise exception as Polystores errors) Feb 17, 2019
tox.ini

README.md

License: MIT Build Status PyPI version Codacy Badge Slack

polystores

Polystores is an abstraction and a collection of clients to interact with cloud storages.

Install

$ pip install -U polystores

N.B. this module does not include by default the cloud storage's client requirements to keep the library lightweight, the user needs to install the appropriate module to use with polystores.

Install S3

pip install boto3 botocore

Install GCS

pip install google-cloud-storage

Install Azure Storage

pip install azure-storage

Stores

This module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.

S3

Normal instantiation

from polystores.stores.s3_store import S3Store

s3_store = S3Store(endpoint_url=..., 
                   access_key_id=...,
                   secret_access_key=...,
                   session_token=...,
                   region=...)

Using env vars

export AWS_ENDPOINT_URL=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SECURITY_TOKEN=...
exprot AWS_REGION=...

And then you can instantiate the store

from polystores.stores.s3_store import S3Store

s3_store = S3Store()

Using a client

from polystores.stores.s3_store import S3Store

s3_store = S3Store(client=client)

Important methods

s3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)
s3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.check_key(key, bucket_name=None)
s3_store.get_key(key, bucket_name=None)
s3_store.read_key(key, bucket_name=None)
s3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)
s3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')
s3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_file(key, local_path, bucket_name=None, use_basename=True)
s3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)

GCS

Normal instantiation

from polystores.stores.gcs_store import GCSStore

gcs_store = GCSStore(project_id=..., 
                     credentials=...,
                     key_path=...,
                     key_path=...,
                     scopes=...)

Using a client

from polystores.stores.gcs_store import GCSStore

gcs_store = GCSStore(client=client)

Important methods

gcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
gcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)
gcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)
gcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)
gcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)

Azure Storage

Normal instantiation

from polystores.stores.azure_store import AzureStore

az_store = AzureStore(account_name=..., 
                      account_key=...,
                      connection_string=...)

Using env vars

export AZURE_ACCOUNT_NAME=...
export AZURE_ACCOUNT_KEY=...
export AZURE_CONNECTION_STRING=...

And then you can instantiate the store

from polystores.stores.azure_store import AzureStore

az_store = AzureStore()

Using a client

from polystores.stores.azure_store import AzureStore

az_store = AzureStore(client=client)

Important methods

az_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
az_store.upload_file(filename, blob, container_name=None, use_basename=True)
az_store.download_file(blob, local_path, container_name=None, use_basename=True)
az_store.upload_dir(dirname, blob, container_name=None, use_basename=True)
az_store.download_dir(blob, local_path, container_name=None, use_basename=True)

Running tests

pytest

License

FOSSA Status