Skip to content
This repository has been archived by the owner. It is now read-only.
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
262 lines (144 sloc) 5.32 KB

Bucket(self, bucket_uri) {#Bucket}

Bucket interface for T4.

__init__

Creates a Bucket object.

Arguments

  • bucket_uri(str): URI of bucket to target. Must start with 's3://'

Returns

A new Bucket

Bucket.config(self, config_url=None) {#Bucket.config}

Updates this bucket's search endpoint based on a federation config.

Bucket.get_user_meta_schema(self) {#Bucket.get_user_meta_schema}

Returns the current search mappings for user metadata from the search endpoint.

Bucket.search(self, query, limit=10) {#Bucket.search}

Execute a search against the configured search endpoint.

Arguments

  • query (str): query string to search
  • limit (number): maximum number of results to return. Defaults to 10

Query Syntax: By default, a normal plaintext search will be executed over the query string. You can use field-match syntax to filter on exact matches for fields in your metadata. The syntax for field match is user_meta.$field_name:"exact_match".

Returns

either the request object (in case of an error) or a list of objects with the following structure:

[{
`"key"`: <key of the object>,
`"version_id"`: <version_id of object version>,
`"operation"`: <"Create" or "Delete">,
`"meta"`: <metadata attached to object>,
`"size"`: <size of object in bytes>,
`"text"`: <indexed text of object>,
`"source"`: <source document for object (what is actually stored in ElasticSeach)>,
`"time"`: <timestamp for operation>,
}...]

Bucket.deserialize(self, key) {#Bucket.deserialize}

Deserializes object at key from bucket.

Arguments

  • key(str): key in bucket to get

Returns

Deserialized object.

Raises

  • KeyError: if key does not exist
  • QuiltException: if deserialization fails in a known way
  • if a deserializer raises an unexpected error

Bucket.__call__(self, key) {#Bucket.__call__}

Deserializes object at key from bucket. Syntactic sugar for bucket.deserialize(key).

Arguments

  • key: Key of object to deserialize.

Bucket.put(self, key, obj, meta=None) {#Bucket.put}

Stores obj at key in bucket, optionally with user-provided metadata.

Arguments

  • key(str): key in bucket to put object to
  • obj(serializable): serializable object to store at key
  • meta(dict): optional user-provided metadata to store

Bucket.put_file(self, key, path, meta=None) {#Bucket.put_file}

Stores file at path to key in bucket.

Arguments

  • key(str): key in bucket to store file at
  • path(str): string representing local path to file Optional args: meta(dict): T4 metadata to attach to file Must be less than 2KiB serialized

Returns

None

Raises

  • if no file exists at path
  • if copy fails

Bucket.put_dir(self, key, directory) {#Bucket.put_dir}

Stores all files in the directory under the prefix key.

Arguments

  • key(str): prefix to store files under in bucket
  • directory(str): path to local directory to grab files from

Returns

None

Raises

  • if directory isn't a valid local directory
  • if writing to bucket fails

Bucket.keys(self) {#Bucket.keys}

Lists all keys in the bucket.

Returns

List of strings

Bucket.delete(self, key) {#Bucket.delete}

Deletes a key from the bucket.

Arguments

  • key(str): key to delete

Returns

None

Raises

  • if delete fails

Bucket.delete_dir(self, path) {#Bucket.delete_dir}

Delete a directory and all of its contents from the bucket.

Arguments

  • path (str): path to the directory to delete

Bucket.ls(self, path=None, recursive=False) {#Bucket.ls}

List data from the specified path.

Arguments

  • path (str): bucket path to list
  • recursive (bool): show subdirectories and their contents as well

Returns

list: Return value structure has not yet been permanently decided Currently, it's a tuple of list objects, containing the following: (directory info, file/object info, delete markers).

Bucket.fetch(self, key, path) {#Bucket.fetch}

Fetches file (or files) at key to path.

If key ends in '/', then all files with the prefix key will match and will be stored in a directory at path.

Otherwise, only one file will be fetched and it will be stored at path.

Arguments

  • key(str): key in bucket to fetch
  • path(str): path in local filesystem to store file or files fetched

Returns

None

Raises

  • if path doesn't exist
  • if download fails

Bucket.get_meta(self, key) {#Bucket.get_meta}

Gets the metadata associated with a key in the bucket.

Arguments

  • key(str): key in bucket to get meta for

Returns

dict of meta

Raises

  • if download fails

Bucket.set_meta(self, key, meta) {#Bucket.set_meta}

Sets user metadata on a key in the bucket.

Arguments

  • key(str): key in bucket to set meta for
  • meta(dict): value to set user metadata to

Returns

None

Raises

  • if put to bucket fails

Bucket.select(self, key, query, raw=False) {#Bucket.select}

Selects data from an S3 object.

Arguments

  • key(str): key to query in bucket
  • query(str): query to execute (SQL by default)
  • query_type(str): other query type accepted by S3 service
  • raw(bool): return the raw (but parsed) response

Returns

pandas.DataFrame: results of query

You can’t perform that action at this time.