This project is no longer supported. A pre-configured collection of tools including Social Feed Manager and Lentil for easily building Twitter and Instagram social media archives on your own computer.
Clone or download

Social Media Combine

The Social Media Combine has been deprecated and is no longer supported.

On January 30th, 2018, Instagram announced the release of a new Instagram Graph API in order to focus on supporting business and advertising use of Instgram content. The Instagram Graph API does not support the use cases required by Lentil. Instagram immediately removed their mechanism for requesting developer credentials for the older Instagram API and it is no longer possible to create new instances of Lentil. The older Instagram API will be deprecated over the next two years. Lentil will continue to work with pre-existing developer credentials until December 11, 2018. Due to the deprecation of Lentil, the Social Media Combine project has also been deprecated and is no longer supported.

The Social Media Combine makes developing collections of social media much easier. It pre-assembles Lentil (for Instagram data harvesting) and Social Feed Manager (for Twitter data harvesting), along with the web servers and databases needed for their use, into a single package that can be deployed to desktop and laptop computers and used in Windows, OSX, or Linux. We believe in the documentary value of social media archives and hope that the Social Media Combine will make it possible to quickly collect social media data on everyday hardware.

What's inside?

  • Social Feed Manager Twitter collecting software from GWU Libraries
  • Lentil Instagram collecting software from NCSU Libraries
  • twarc Twitter collecting software developed by Ed Summers
  • A full pre-configured server environment to support all included software
  • A new single web-based configuration interface for all included software
  • Upgrade and system management scripts
  • Some data export capabilities

Steps to run

  1. Install Git

  2. Install Vagrant version 1.7.4 or greater on your host machine.

  3. Open Terminal on a Mac, Git Shell on Windows, or your preferred shell on Linux and clone this repository using git.

    git clone
    cd Social-Media-Combine
  4. You can run Combine on VM (step 4.1) OR Combine without VM (directly on Docker)(step 4.2).

4.1. Combine on VM

  • Install Virtualbox
  • Run vagrant up from within the Social-Media-Combine directory.

4.2. Combine without VM [tested on Ubuntu 14.04]

  • Install Docker version 1.8.1 or greater

  • Run Docker Daemon on public interface

    Append following to the /etc/default/docker file

     DOCKER_OPTS="-H <ip>:<port>"

    Restart docker service

     service docker restart

    Define DOCKER_HOST and VAGRANT_NO_PARALLEL environment variables

     export DOCKER_HOST=<ip>:<port>
     export VAGRANT_NO_PARALLEL=true
  • Run

    vagrant up --docker=true

If working on Windows and vagrant up immediately returns errors, you may need to run sh

This opens a web browser with configuration form. If it doesn't, visit <http://localhost:8081> in your browser. Make changes and click OK.
  1. Enter configuration parameters in the web configuration form and click OK. Your applications should be updated within a minute or so.


To stop the Social Media Combine virtual machine, run vagrant halt in the project directory.


  1. The first time it's started, the Combine will download a lot of application data. Occasionally, one of these downloads will fail and you may see some error messages or be unable to reach one of the applications. In order to retry the Combine setup, run vagrant provision.
  2. Don't see any data? Lentil will harvest images every 15 minutes, and Social Feed Manager will harvest Tweets every 6 hours.

Exporting data

After you have been harvesting content for a day or so, take a look in the archive directory. Some Social Feed Manager content may not yet appear in this directory, but will still be present in the database.

To generate a CSV export of data collected for a Twitter user (e.g. @cazzerson) in Social Feed Manager, you can run vagrant exec 'docker exec vagrant_sfmapp_1 ./sfm/ export_csv --twitter-user cazzerson'. You can also add users to a set (e.g. myset) in the Social Feed Manager administrative tools and then export this set of users with vagrant exec 'docker exec vagrant_sfmapp_1 ./sfm/ export_csv --set-name myset'.

Administrative information


Social Feed Manager:

Steps for Reconfiguration

Application reconfiguration can be performed after deployment.

  1. Open web configuration tool at http://localhost:8081. Make changes and click OK.

  2. Your applications should be updated within a minute or so.

If your applications are not updated or you are changing configuration files on the filesystem, you may need to manually launch the configuration task by running vagrant exec ./

Steps for updating applications

  1. In your terminal application, while in the Social Media Combine directory, run
    vagrant provision


Please email Bret Davidson at with any questions or suggestions.


  • Upgrade and Rollback
    • There is a hook in Vagrantfile for upgrade procedure.
    • The script shall perform check point to roll back in case of failure.
    • Then check the version number of lentil and Social Feed Manager application. Perform upgrade according to specified version.
    • If possible take the target version from web config form.
  • Provide more configurations to Social Feed Manager container. As of now we have only 3 parameters for Social Feed Manager.
  • Run webconfig container on demand. Start the container only when required.
  • Enable automatic port selection in Vagrantfile. Automatically change the webconfig port 8081 to auto selected port.
  • Give a host name to each application instead of localhost and port number.

Developer Notes

Docker Documentation


  • Aditya Ambre
  • Jason Casden
  • Bret Davidson
  • Brian Dietz
  • Abigail Jones
  • Ramakant Moka
  • Mitul Panchal
  • Our many testers

The development of the Social Media Combine environment was partially supported by federal Library Services and Technology Act (LSTA) funds made possible through a grant from the Institute of Museum and Library Services, and administered by the State Library of North Carolina, a division of the Department of Cultural Resources. This grant-funded project, "New Voices and Fresh Perspectives," is hosted by NCSU Libraries and led by Brian Dietz and Jason Casden.

Social Feed Manager is developed at George Washington University Libraries.

Lentil is developed at NCSU Libraries.

twarc is developed by Ed Summers.