Merge branch 'master' of https://github.com/berkmancenter/mediacloud …

…into zh_support

.travis.yml

@@ -2,7 +2,7 @@
 sudo: true
 dist: trusty
 addons:
-    postgresql: "9.3"
+    postgresql: "9.6"
     hosts:
         - mediacloud.local
 language: generic

doc/hosting/postgres-hosting.markdown

@@ -28,7 +28,6 @@ hroberts@mcdb1:~$ pps
  11059 | mediacloud | active | 2016-05-25 12:53:40.141334-04 | select stories_id from stories where url in ($1)
   7380 | mediacloud | active | 2016-05-25 12:53:39.948636-04 | INSERT INTO queue (priority, unique_key, function_name, d
  11042 | mediacloud | active | 2016-05-25 12:53:39.947215-04 | select stories_id from stories where url in ($1)
- 11757 | mediacloud | active | 2016-05-25 12:53:39.906391-04 |  SELECT upsert_bitly_clicks_total($1, $2)
   9689 | PostgreSQL   | active | 2016-05-25 09:56:49.546405-04 | autovacuum: VACUUM public.story_sentences (to prevent wra
  ```
 

doc/vagrant.markdown

@@ -1,5 +1,4 @@
-Media Cloud on Vagrant
-======================
+# Media Cloud on Vagrant
 
 You might want to run Media Cloud with [Vagrant](http://www.vagrantup.com/) in the following cases:
 
@@ -11,8 +10,7 @@ You might want to run Media Cloud with [Vagrant](http://www.vagrantup.com/) in t
 Media Cloud provides a lean `Vagrantfile` script located in `script/vagrant/` that you can use to set up a Media Cloud instance on Vagrant.
 
 
-Installing Vagrant and its dependencies
----------------------------------------
+## Installing Vagrant and its dependencies
 
 Download and install Vagrant 1.3+ from the [Vagrant Downloads page](http://downloads.vagrantup.com/). APT's version is too old at the time of writing.
 
@@ -22,30 +20,35 @@ You'll also need a Provider -- `x86_64` virtualization software package / servic
 * If you would like to set up and run Media Cloud instances remotely, use [Amazon EC2](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1).
 
 
-Setting up Media Cloud on Vagrant
----------------------------------
+## Setting up Media Cloud on Vagrant
 
-Please note that the initial process of setting up Media Cloud may take a long time (~2 hours) as Vagrant will go about upgrading packages, installing a lot of dependencies, compiling and testing Perl, testing Media Cloud itself, etc.
+Please note that the initial process of setting up Media Cloud may take a long time (up to 2 hours) as Vagrant will go about upgrading packages, installing a lot of dependencies, compiling and testing Perl, testing Media Cloud itself, etc.
 
+Before running Media Cloud on one of Vagrant's providers (VirtualBox or EC2), do the following:
 
-### On VirtualBox
-
-To create a new Media Cloud instance on Vagrant using the VirtualBox provider:
+    # Windows: check out submodules with LF line endings
+    git config --global core.eol lf
+    git config --global core.autocrlf input
 
-    # (Run once) Add Ubuntu 16.04 as a Vagrant box for later reuse
-    vagrant box add ubuntu/xenial64
+    # Windows: enable Git symlinks
+    git config --global config.symlinks true
 
     # Check out a *fresh* copy of Media Cloud from the Git repository
-    git clone --recursive https://github.com/berkmancenter/mediacloud.git vagrant-mediacloud/
+    git clone https://github.com/berkmancenter/mediacloud.git vagrant-mediacloud/
     cd vagrant-mediacloud/
+    git submodule update --init --recursive
 
     # Change to the directory in which "Vagrantfile" and other files are located
     cd script/vagrant/
 
+
+### On VirtualBox
+
+To create a new Media Cloud instance on Vagrant using the VirtualBox provider:
+
     # Power on the virtual machine
     #
-    # (Note: if running from Windows, "vagrant up" will require to be run with
-    # Administrator privileges and Git's "config.symlinks" set to "true")
+    # Windows: "vagrant up" is required to be run with Administrator privileges to be able to mount SMB shared folder
     vagrant up --provider=virtualbox
 
 
@@ -54,19 +57,15 @@ To create a new Media Cloud instance on Vagrant using the VirtualBox provider:
 To create a new Media Cloud instance on Vagrant using the Amazon EC2 provider:
 
 1. Find out the AWS credentials to be used by Vagrant:
-    * `AWS_ACCESS_KEY_ID`:
-        * Access Key ID.
-        * Available either at the [Your Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential) or the [Security Credentials](https://portal.aws.amazon.com/gp/aws/securityCredentials) pages.
-    * `AWS_SECRET_ACCESS_KEY`:
-        * Secret Access Key.
-        * Available either at the [Your Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential) or the [Security Credentials](https://portal.aws.amazon.com/gp/aws/securityCredentials) pages.
-    * `AWS_KEYPAIR_NAME` and `AWS_SSH_PRIVKEY`:
+    * `AWS_ACCESS_KEY_ID` -- Access Key ID; available either at the [Your Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential) or the [Security Credentials](https://portal.aws.amazon.com/gp/aws/securityCredentials) pages.
+    * `AWS_SECRET_ACCESS_KEY` -- Secret Access Key; available either at the [Your Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential) or the [Security Credentials](https://portal.aws.amazon.com/gp/aws/securityCredentials) pages.
+    * `AWS_KEYPAIR_NAME` and `AWS_SSH_PRIVKEY` -- SSH keypair for accessing the instance:
         1. Go to the [EC2 - Key Pairs](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:) page.
         2. Click "Create Key Pair", name it `development`.
         3. Download `development.pem`, save it as `~/path/to/development.pem`.
         4. `AWS_KEYPAIR_NAME="development"`
         5. `AWS_SSH_PRIVKEY="~/path/to/development.pem"`
-    * `AWS_SECURITY_GROUP`:
+    * `AWS_SECURITY_GROUP` -- security group to put the instance to:
         1. Go to the [EC2 - Security Groups](https://console.aws.amazon.com/ec2/home?region=us-east-1#s=SecurityGroups) page.
         2. Click "Create Security Group", name it `default`.
         3. Allow "Inbound SSH traffic" from `0.0.0.0/0`.
@@ -79,13 +78,6 @@ To create a new Media Cloud instance on Vagrant using the Amazon EC2 provider:
         # (Run once) Install the "vagrant-aws" plugin
         vagrant plugin install vagrant-aws
 
-        # Check out a *fresh* copy of Media Cloud from the Git repository
-        git clone --recursive https://github.com/berkmancenter/mediacloud.git vagrant-mediacloud/
-        cd vagrant-mediacloud/
-
-        # Change to the directory in which "Vagrantfile" and other files are located
-        cd script/vagrant/
-
         # (Run once) Add the "dummy" AWS box for later reuse
         vagrant box add ubuntu_aws aws_ec2_dummy.box
 
@@ -101,8 +93,7 @@ To create a new Media Cloud instance on Vagrant using the Amazon EC2 provider:
         vagrant up --provider=aws
 
 
-Running Media Cloud on Vagrant
-------------------------------
+## Running Media Cloud on Vagrant
 
 The Git repository that you have cloned previously is automatically mounted to `/mediacloud` directory on the virtual machine.
 
@@ -167,8 +158,7 @@ To start the Media Cloud web service:
     * For example, if the Public DNS of the EC2 instance is `ec2-54-224-57-211.compute-1.amazonaws.com`, access the web service by opening `http://ec2-54-224-57-211.compute-1.amazonaws.com:3000/`.
 
 
-Testing Media Cloud with Vagrant
---------------------------------
+## Testing Media Cloud with Vagrant
 
 Directory `script/vagrant/` contains a script `run_install_test_suite_on_vagrant.sh` that:
 

install/install_modules_with_carton.sh

@@ -20,10 +20,69 @@ else
     OPENSSL_PREFIX="/usr"
 fi
 
-if [ `uname` == 'Darwin' ]; then
-    # needed to install the https github links below
-    ./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm LWP::Protocol::https
-fi
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    Path::Class
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    Path::Class
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    Test::Fatal File::Listing HTTP::Cookies HTTP::Daemon HTTP::Negotiate Net::HTTP WWW::RobotRules Net::SSLeay
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    Test::Fatal File::Listing HTTP::Cookies HTTP::Daemon HTTP::Negotiate Net::HTTP WWW::RobotRules Net::SSLeay
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    Test::RequiresInternet
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    Test::RequiresInternet
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    IO::Socket::SSL
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    IO::Socket::SSL
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    LWP::UserAgent
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    LWP::UserAgent
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    Net::SSL
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    Net::SSL
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    LWP::Protocol::https
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local/ \
+    --verbose \
+    LWP::Protocol::https
+
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --verbose \
+    Net::HTTP
+./script/run_with_carton.sh ~/perl5/perlbrew/bin/cpanm \
+    --local-lib-contained local \
+    --verbose \
+    Net::HTTP
 
 # OS X no longer provides OpenSSL headers and Net::AMQP::RabbitMQ doesn't care
 # about OPENSSL_PREFIX so we need to set CCFLAGS and install the module
@@ -49,7 +108,7 @@ fi
 # Inline::Python needs to use virtualenv's Python 3 instead of the default Python2
 set +u; source mc-venv/bin/activate; set -u
 INLINE_PYTHON_EXECUTABLE=`command -v python3`   # `which` is a liar
-echo "Installing Inline::Python with this python executable: $INLINE_PYTHON_EXECUTABLE"
+echo "Installing Inline::Python with Python executable: $INLINE_PYTHON_EXECUTABLE"
 
 # Install Inline::Python variant which die()s with tracebacks (stack traces)
 INLINE_PYTHON_EXECUTABLE=$INLINE_PYTHON_EXECUTABLE \

install/install_postgresql_server_packages.sh

@@ -49,7 +49,11 @@ else
     echo | sudo ./install/setup_postgresql_apt.sh
     # assume Ubuntu
     sudo apt-get --assume-yes install \
-        postgresql postgresql-client postgresql-contrib postgresql-server-dev-all
+        postgresql-9.6 \
+        postgresql-client-9.6 \
+        postgresql-contrib-9.6 \
+        postgresql-plperl-9.6 \
+        postgresql-server-dev-all
 
 fi