Publish data easily, quickly and correctly
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github
app Add Google Analytics script to head of application.html.erb May 23, 2018
bin Add rspec config for spring, else it doesn't work Jan 26, 2017
config Add privacy policy view, route and link_to on footer navigation layout May 23, 2018
db Revert "Bugs" Mar 19, 2018
extra
lib fix populate task Sep 22, 2017
log Revert "Bugs" Mar 19, 2018
public Replace column with warning favicon Apr 25, 2017
spec
vendor/assets Add UUID little JS library Feb 14, 2017
.gitignore Add byebug for testing Aug 8, 2017
.rspec
.ruby-version
.simplecov
.travis.yml Revert "Bugs" Mar 19, 2018
CODE_OF_CONDUCT.md
CONTRIBUTING.md
Gemfile Revert "Bugs" Mar 19, 2018
Gemfile.lock GEM updates - loofah, nokogiri, rack-protection, rails-html-sanitizer Mar 29, 2018
Guardfile
LICENSE.md Bring in the Jekyll improvements (#177) Aug 11, 2016
Procfile Use Puma instead of Unicorn (#181) Aug 11, 2016
README.md
Rakefile Blank Rails project Oct 6, 2014
app.json Revert "Bugs" Mar 19, 2018
config.ru
dependencyci.yml API changes (#173) Aug 11, 2016

README.md

Build Status Dependency Status Coverage Status Code Climate License Dependency Status Badges

Octopub

Octopub is a Ruby on Rails application that provides a simple and frictionless way for users to publish data easily, quickly and correctly on GitHub.

Summary of features

More information is in the announcement blog post

The live instance of Octopub is running at http://octopub.io/

Follow the public feature roadmap for Octopub

Requirements

These are the tools and services required to get Octopub fully working for development, testing and production environments. We'll explain how to set these up in the next section.

Setup

Redis/Sidekiq

Sidekiq is used for managing the background proccessing of data uploads. To use Sidekiq just install Redis by following the instructions here, or if you are using Homebrew you can just do brew install redis.

Environment variables

For development Octopub uses the dotenv gem to load environment variables. Create a file called .env in your project root and paste in the variables below. We'll fill these in as we go along.

# GitHub App Client ID & secret
GITHUB_KEY=
GITHUB_SECRET=

# OAuth access token for GitHub API access
GITHUB_TOKEN=

S3_BUCKET=

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

PUSHER_APP_ID=
PUSHER_KEY=
PUSHER_SECRET=
PUSHER_CLUSTER=

BASE_URI=
ODC_API_KEY=
ODC_USERNAME=

# production only
SMTP_USERNAME=
SMTP_PASSWORD=
SMTP_SERVER=

GitHub

Create a GitHub application:

  1. Log in to GitHub.
  2. In Settings -> Developer settings -> OAuth applications, create a new OAuth application with a unique name. You can use http://octopub.io for the homepage and for the callback URL use your local server address, i.e. http://localhost:3000. Click on your OAuth application to see your Client ID and Client Secret, and update your .env file:
GITHUB_KEY=<YOUR CLIENT ID>
GITHUB_SECRET=<YOUR CLIENT SECRET>
  1. In Settings -> Developer settings -> Personal access tokens, generate a new token with a sensible description, e.g. octopub_dev_token, and update your .env file:
GITHUB_TOKEN=<Your token>

AWS

Create an S3 bucket:

  1. In AWS go to the S3 service and create a bucket with a sensible name. Make sure the region is set to EU(Ireland) since Octopub uses this.
  2. Click on your bucket and go to the Permissions tab. Click on CORS Configuration and paste in the configuration below. This will allow your local development version of Octopub to make requests to your S3 bucket.
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>http://localhost:3000</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Grant permissions to your bucket:

  1. In AWS go to the IAM (Identity and Access Management page) service.
  2. Click Users.
  3. Add a new user and give it a name, e.g. octopub-development, and for Access Type select Programmatic Access.
  4. For permissions, select Attach existing policies directly - this will open a new tab in your browser.
  5. Click create your own policy and give it a name, e.g. octopub-dev-permissions. Then for the policy document, use the following template, but add your bucket name in place of <BUCKETNAME>.
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "AllowAdminAccessToBucketOnly",
           "Action": [
               "s3:*"
           ],
           "Effect": "Allow",
           "Resource": [
               "arn:aws:s3:::<YOUR BUCKET NAME>",
               "arn:aws:s3:::<YOUR BUCKET NAME>/*"
           ]
       }
   ]
}
  1. Click validate policy just to be sure you've not made a typo, then confirm.
  2. Back on the Set permissions page, select the policy you've just created in the table by selecting the checkbox, then click Review and then click Create user.
  3. Download the CSV file containing your Access key ID and Secret access key and update your .env file:
AWS_ACCESS_KEY_ID=<YOUR ACCESS KEY ID>
AWS_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>
S3_BUCKET=<YOUR BUCKET NAME>

Pusher

  1. Log in to https://pusher.com or create a free account.
  2. Create a new application and call it something sensible.
  3. Select the App Keys tab and use the relevant values there to update your .env file:
PUSHER_APP_ID=
PUSHER_KEY=
PUSHER_SECRET=
PUSHER_CLUSTER=

ODC (open data certificate) setup

  1. Log in to https://certificates.theodi.org/ or create a free account.
  2. Go to your profile page, copy your API token and update your .env file:
ODC_API_KEY=<API TOKEN>
ODC_USERNAME=<YOUR USERNAME (email address you used when signing up)>

Running the full application locally

Assuming you have completed the setup instructions above...

  • Start Redis with redis-server.
  • Start Sidekiq with bundle exec sidekiq in the application directory.
  • Create the postgresql databases specified in config/database.yml and run rails db:migrate.
  • Start Octopub with rails s in the application directory.
  • Navigate to the home page.
  • Sign into octopub with your GitHub account.

Congratulations, you should be signed in! Now try adding some data.

Checking the Sidekiq queue

Start a rails console session and then...

require 'sidekiq/api'
Sidekiq::Queue.new.size
Sidekiq::Queue.new.first

Tests

Octopub uses the rspec test framework and requires the presence of a .env. See earlier section for details as you can (re)use your development variables*

The test suite can be run with bundle exec rspec.

* Note - the tests use VCR or mocking to allow the tests to be run offline without interfacing with the services.

Deployment

A commit to master will trigger a TravisCI run; If successful it will automatically deploy to Heroku.

Caching

The GitHub organisations are cached for the logged in user. They can be cleared from a console with Rails.cache.clear