s3: provide a way to use key id and key secret directly #4175
Description
For now the only way to pass credentials to S3 is using credentials file and the env var. This is very inconvenient when trying to use dvc as a library. For now I hacked it like:
def pull(repo):
# In memory config update to avoid validation
repo.config["core"]["remote"] = "vdefault"
repo.config["remote"]["vdefault"] = {"url": ..., "key_id": ..., "key_secret": ...}
fix_s3()
repo.pull()
from funcy import once, monkey, cached_property, wrap_prop # noqa
@once
def fix_s3():
import threading
from dvc.remote.s3 import S3RemoteTree
@monkey(S3RemoteTree, "s3")
@wrap_prop(threading.Lock())
@cached_property
def s3(self):
import boto3
session = boto3.session.Session(
profile_name=self.profile,
region_name=self.region,
# Pass these two from config to session
aws_access_key_id=self.config.get("key_id"),
aws_secret_access_key=self.config.get("key_secret"),
)
return session.client("s3", endpoint_url=self.endpoint_url, use_ssl=self.use_ssl)Would be nice to have it inside dvc.remote.S3RemoteTree.