Merge 7f848f7 into 7592cf1

.gitignore

@@ -4,6 +4,8 @@
 *.egg-info
 .cache
 build/
+build-aws/
 dist/
+dist-aws/
 tests/db/
 .pytest_cache

.travis.yml

@@ -2,13 +2,27 @@ sudo: false
 notifications:
   email: false
 language: python
-matrix:
-  include:
-    - python: 3.6
-      env: TOX_ENV=py36
-    - python: 3.6
-      env: TOX_ENV=coveralls
+python:
+  - "3.6"
+env:
+  - TOX_ENV=py36
+  - TOX_ENV=coveralls
 install:
   - pip install tox
 script:
   - tox -e $TOX_ENV
+jobs:
+  include:
+    - stage: s3 deploy
+      install:
+        - pip install pipenv awscli
+      script: skip
+      before_deploy: bash build.sh
+      deploy:
+        skip_cleanup: true
+        provider: s3
+        access_key_id: $AWS_ACCESS_KEY_ID
+        secret_access_key: $AWS_SECRET_ACCESS_KEY
+        bucket: "carbon-deploy"
+        local_dir: "dist-aws"
+      after_deploy: bash publish.sh

Makefile

@@ -7,7 +7,7 @@ all: test
 clean:
 	find . -name "*.pyc" -print0 | xargs -0 rm -f
 	find . -name '__pycache__' -print0 | xargs -0 rm -rf
-	rm -rf .coverage .tox *.egg-info .eggs
+	rm -rf .coverage .tox *.egg-info .eggs build/ build-aws/ dist/ dist-aws/
 
 install:
 	pipenv install

Pipfile

@@ -15,3 +15,4 @@ pyopenssl = "*"
 click = "*"
 lxml = "*"
 SQLAlchemy = "*"
+cx-oracle = "*"

README.rst

@@ -1,35 +1,57 @@
 carbon
 ======
 
-Carbon is a tool for generating a feed of people that can be loaded into Symplectic Elements.
+Carbon is a tool for generating a feed of people that can be loaded into Symplectic Elements. It is run as a Lambda by a scheduled CloudWatch event.
 
+Developing
+----------
 
-Installation
-------------
+Use pipenv to install and manage dependencies::
 
-In order to connect to the data warehouse you will need to install the ``cx_Oracle`` python package into your virtualenv. Steps 1-5 detailed at https://blogs.oracle.com/opal/entry/configuring_python_cx_oracle_and should be sufficient.
+    $ git clone git@github.com:MITLibraries/carbon.git
+    $ cd carbon
+    $ pipenv install --dev
 
-Once the Oracle package has been installed clone the repo::
+Connecting to the data warehouse will require installing the ``cx_Oracle`` python package. The good news is that this is now being packaged as a wheel for most architectures, so no extra work is required to install it. If you don't need to actually connect to the data warehouse, you are done. Note that the test suite uses SQLite, so you can develop and test without connecting to the data warehouse.
 
-    (carbon)$ git clone https://github.com/MITLibraries/carbon.git
+If you do need to connect to the data warehouse, you will also need to install the Oracle client library. It seems that now just installing the basic light package should be fine. In general, all you should need to do is extract the package and add the extracted directory to your ``LD_LIBRARY_PATH`` environment variable. If there is no ``lbclntsh.so`` (``libclntsh.dylib`` for Mac) symlink in the extracted directory, you will need to create one. The process will look something like this::
 
-Then use pip to install::
+    $ unzip instantclient-basiclite-linux.x64-18.3.0.0.0dbru.zip -d /usr/local/opt
+    # Add the following line to your .bash_profile or whatever to make it permanent
+    $ export LD_LIBRARY_PATH=/usr/local/opt/instantclient_18_3:$LD_LIBRARY_PATH
+    # If the symlink doesn't already exist:
+    $ ln -rs /usr/local/opt/instantclient_18_3/libclntsh.so.18.1 \
+        /usr/local/opt/instantclient_18_3/libclntsh.so
 
-    (carbon)$ cd carbon && pip install .
+On Linux, you will also need to make sure you have libaio installed. You can probably just use your system's package manager to install this easily. The package may be called ``libaio1``.
 
+The Lambda Package
+------------------
 
-Updating
---------
+The Lambda package is built with the ``build.sh`` script.
 
-To update carbon, update the local repo and then upgrade the local package::
+**IMPORTANT**: Both the ``cx_Oracle`` and ``lxml`` packages use platform dependent wheels, so you if you are planning on using the Lambda package you *must* do this step on a Linux x86_64 architecture.
 
-    (carbon)$ cd carbon && git pull
-    (carbon)$ pip install -U .
+We are restricted from distributing the Oracle client library, so a copy is kept in a private S3 bucket for use when building the Lambda. The Lambda execution context does not provide libaio, so a copy of this is also kept in the bucket. If you are updating either of these, make sure you are using a Linux x86_64 version. To be on the safe side, use https://github.com/lambci/docker-lambda to get the libaio library as that will ensure you are getting one that's been linked against an appropriate version of glibc.
 
+The build process downloads each of these files from S3 so you should have the AWS CLI installed and configured to authenticate using an account with appropriate access.
+
+Deploying
+---------
+
+Deployment is currently being handled by Travis. When a PR is merged onto the master branch Travis will build a new Lambda package, push it to S3 and update the Lambda function to point to the new package.
+
+If you need to deploy a new package outside of Travis then do the following, *noting the restrictions on running build.sh described above*::
+
+    $ cd carbon
+    $ ./build.sh
+    $ ./publish.sh
 
 Usage
 -----
 
+While this is intended to be run as a Lambda, the old CLI interface is still supported for ease of testing locally.
+
 View the help menu for the ``carbon`` command::
 
     $ carbon --help

build.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -e
+
+BUILD_DIR=build-aws
+DIST_DIR=dist-aws
+S3_BUCKET=carbon-deploy
+LIBAIO_SO=libaio.so.1.0.1
+ORACLE_ZIP=instantclient-basiclite-linux.x64-18.3.0.0.0dbru.zip
+PACKAGE=carbon-`git rev-parse --short HEAD`.zip
+
+mkdir -p $BUILD_DIR/lib
+mkdir -p $DIST_DIR
+
+aws s3 cp s3://$S3_BUCKET/$ORACLE_ZIP $BUILD_DIR/$ORACLE_ZIP && \
+  unzip -j $BUILD_DIR/$ORACLE_ZIP -d $BUILD_DIR/lib/ 'instantclient_18_3/*' && \
+  rm $BUILD_DIR/$ORACLE_ZIP
+aws s3 cp s3://$S3_BUCKET/$LIBAIO_SO $BUILD_DIR/lib/$LIBAIO_SO && \
+  ln -rs $BUILD_DIR/lib/$LIBAIO_SO $BUILD_DIR/lib/libaio.so.1 && \
+  ln -rs $BUILD_DIR/lib/libaio.so.1 $BUILD_DIR/lib/libaio.so
+cp -r carbon $BUILD_DIR
+cp lambda.py $BUILD_DIR
+pipenv lock -r > $BUILD_DIR/requirements.txt
+pipenv run pip install -r $BUILD_DIR/requirements.txt -t $BUILD_DIR
+cd $BUILD_DIR && zip --symlinks -r ../$DIST_DIR/$PACKAGE *

publish.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -e
+
+PACKAGE=carbon-`git rev-parse --short HEAD`.zip
+S3_BUCKET=carbon-deploy
+
+aws lambda update-function-code --function-name carbon-test --s3-bucket \
+  $S3_BUCKET --s3-key $PACKAGE --region us-east-1