aws/AWS_Boto3_s3_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS S3 using boto3\n",
    "This tutorial will demonstrate how to interact with S3 bucket from python using boto3 library.\n",
    "For ad hoc one time operations you may find the AWS [command line interface](https://aws.amazon.com/cli/) more useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to get programtic access to aws you will need to provide a `AWS Access Key ID` and `AWS Secret Access Key` even if the resource is \"public\"\n",
    " * The easiest way to authenticate yourself is to install the AWS CLI mentioned above and run command `aws configure` from the command line.\n",
    " * When you run `aws configure` it should create a file in `~/.aws/credentials` which contains the login credentials. Boto3 will automatically recognize credentails stored in this location.\n",
    " * If you are trying to access a bucket owned by another team/person you will need to have them provide you with the crednetials. AWS configure allows you to setup each new credential under a profile. You can tell boto3 which credential you'd like to use by passing the `profile` argument otherwise it will pickup the `default` profile (if existing).\n",
    " * Alternatively, you can type your credentials directly into the script but this is not recommended"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please read through [this](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html) guide to setup your credentials. For this tutorial I am using my own personal test bucket. If you'd like to actually follow the steps you will need your own credentials and a test bucket. If you'd like to use my bucket to test contact @bstarling in slack and I can provide you with credentials to access `public-test-bucket-d4d` used in this tutorial**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['d4d', 'd4d_tutorial', 'd4d_s3', 'dynro', 'default']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that boto3 is able to pick up valid credentials\n",
    "# Will print a list of profiles\n",
    "session = boto3.Session(profile_name='d4d_tutorial')\n",
    "session.available_profiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Tell boto3 which resource you will use\n",
    "s3 = session.resource('s3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Specify AWS bucket\n",
    "bucket = s3.Bucket('public-test-bucket-d4d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3.ObjectSummary(bucket_name='public-test-bucket-d4d', key='tutorial/')\n",
      "s3.ObjectSummary(bucket_name='public-test-bucket-d4d', key='tutorial/data.csv')\n",
      "s3.ObjectSummary(bucket_name='public-test-bucket-d4d', key='tutorial/file_one.txt')\n",
      "s3.ObjectSummary(bucket_name='public-test-bucket-d4d', key='tutorial/file_three.txt')\n",
      "s3.ObjectSummary(bucket_name='public-test-bucket-d4d', key='tutorial/file_two.txt')\n"
     ]
    }
   ],
   "source": [
    "# print all objects in bucket\n",
    "for obj in bucket.objects.all():\n",
    "    print(obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3.Object(bucket_name='public-test-bucket-d4d', key='tutorial/data.csv')\n"
     ]
    }
   ],
   "source": [
    "# get object by name\n",
    "file = bucket.Object(key='tutorial/data.csv')\n",
    "print(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AcceptRanges': 'bytes',\n",
       " 'Body': <botocore.response.StreamingBody at 0x108fa0e48>,\n",
       " 'ContentLength': 8,\n",
       " 'ContentType': 'text/csv',\n",
       " 'ETag': '\"8015171fe51e613df5dcdf8e89e94b1c\"',\n",
       " 'LastModified': datetime.datetime(2017, 1, 24, 14, 14, 45, tzinfo=tzutc()),\n",
       " 'Metadata': {},\n",
       " 'ResponseMetadata': {'HTTPHeaders': {'accept-ranges': 'bytes',\n",
       "   'content-length': '8',\n",
       "   'content-type': 'text/csv',\n",
       "   'date': 'Wed, 25 Jan 2017 12:26:49 GMT',\n",
       "   'etag': '\"8015171fe51e613df5dcdf8e89e94b1c\"',\n",
       "   'last-modified': 'Tue, 24 Jan 2017 14:14:45 GMT',\n",
       "   'server': 'AmazonS3',\n",
       "   'x-amz-id-2': 'MfNDWsYCnrUpZPsw6ccST05Tb3CK2l4GR3MP8HenBLEkaSlEmkW4ScgKQgRaBIexSBFkz5dPmWo=',\n",
       "   'x-amz-request-id': '4DD3D2C04D199104'},\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HostId': 'MfNDWsYCnrUpZPsw6ccST05Tb3CK2l4GR3MP8HenBLEkaSlEmkW4ScgKQgRaBIexSBFkz5dPmWo=',\n",
       "  'RequestId': '4DD3D2C04D199104',\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# AWS s3 object\n",
    "file.get()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'my data\\n'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read file body (careful doing this with large files)\n",
    "file.get()['Body'].read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Download file\n",
    "s3.meta.client.download_file('public-test-bucket-d4d', 'tutorial/data.csv', 'local_data_file.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# or using attributes of variables assigned earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s3.meta.client.download_file(bucket.name, file.key, 'local_data_new.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#download all files in a s3 \"folder\" with specific prefix:\n",
    "for item in bucket.objects.filter(Prefix='tutorial/file'):\n",
    "    s3.meta.client.download_file(bucket.name, item.key, 'local_{}'.format(item.key.split('/')[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Check dir contents after download\n",
    "%ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE: unless you need to do you work inside python you can do a lot with the [AWS CLI](http://docs.aws.amazon.com/cli/latest/reference/s3/index.html) which uses botocore under the hood*\n",
    "* **`aws s3 sync <local_folder> s3://bstarling-d4d/data/test`**  sync a folder (need read/write access)\n",
    "* **`aws s3 cp s3://public-test-bucket-d4d/tutorial/data.csv local_data.csv`** copy a file from s3 to local file\n",
    "* **`aws s3 cp local_data.csv s3://public-test-bucket-d4d/tutorial/data.csv`** push local file to s3\n",
    "* **`aws s3 ls s3://public-test-bucket-d4d/tutorial/ --profile d4d_tutorial`** list contents (using d4d_tutorial credentials)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}