Ruby HTML Gherkin CSS JavaScript CoffeeScript
Clone or download
Permalink
Failed to load latest commit information.
.github Consistent nomenclature - improve > enhance Jul 28, 2017
app adding links to ICO May 23, 2018
bin Basic rails 4.0.2 / Ruby 2.1 app Jan 7, 2014
config Update airbrake config Jul 8, 2017
db init schema Jan 7, 2014
features Include steps to eval row numbers Apr 25, 2017
fixtures Addition of fixtures for stress testing Apr 25, 2017
lib Money patch DataKitten to remove linked_data dependency Nov 17, 2016
log Basic rails 4.0.2 / Ruby 2.1 app Jan 7, 2014
public This was missing Dec 4, 2015
script cuke it up Jan 7, 2014
spec fix tag output string in test Apr 26, 2017
vendor/assets Basic rails 4.0.2 / Ruby 2.1 app Jan 7, 2014
.coveralls.yml Might as well add this too Jan 10, 2014
.gitignore Merge branch 'master' of github.com:theodi/csvlint into feature-suppo… Mar 17, 2014
.powrc Add pow config Jan 9, 2014
.rspec fix rspec installation for rails Jan 9, 2014
.ruby-version update to ruby 2.4.1 Apr 26, 2017
.travis.yml update to ruby 2.4.1 Apr 26, 2017
CODE_OF_CONDUCT.md Refactor .github to only GH relevant MD files Jul 28, 2017
CONTRIBUTING.md Refactor .github to only GH relevant MD files Jul 28, 2017
Gemfile Bump rake from 12.0.0 to 12.3.0 Nov 16, 2017
Gemfile.lock Bump airbrake from 7.1.0 to 7.1.1 Dec 22, 2017
Guardfile Add guard Jan 7, 2014
LICENSE.md Badges and license Jan 7, 2014
Procfile Add Puma to procfile Nov 11, 2016
README.md remove incorrect link Jul 5, 2017
Rakefile Not in production Jan 9, 2014
app.json Add Pusher to app.json Nov 15, 2016
byebug.md removing schema_data and changing to schema file on this line because… Jul 9, 2015
config.ru Basic rails 4.0.2 / Ruby 2.1 app Jan 7, 2014

README.md

Build Status Coverage Status Code Climate Dependency Status License

CSVLint

CSVlint is an online validation tool for CSV files. It validates conformity of CSV releases to standards, checks for missing or malformed data, and can validate against both CSVW and Datapackage schema standards.

Summary of features

CSVlint is a rails app designed to act as a continuous validation service, so that when data is changed online, the validation is updated. Data can be online or uploaded for private validation.

The validation code is actually all done in a Ruby gem that can also be freely reused in other projects

Follow the public feature roadmap for CSVLint

Development

Requirements

Ruby version 2.3.1

The application uses mongod and redis databases as background jobs for data persistence

.env file (see below)

Environment variables

For running tests:

PUSHER_APP_ID
PUSHER_KEY
PUSHER_SECRET
PUSHER_CLUSTER

For development add:

AWS_ACCESS_KEY
AWS_BUCKET_NAME
AWS_SECRET_ACCESS_KEY

In production add:

  • REDIS_PROVIDER
  • CSVLINT_SESSION_SECRET

Setting up the required environment variables

Pusher setup
  1. Log in to https://pusher.com
  2. Create a new application and call it something sensible
  3. Select the App Keys tab and get the values and paste them in to your .env file
PUSHER_APP_ID=
PUSHER_KEY=
PUSHER_SECRET=

NOTE: You may be set up for a non-default Pusher cluster (The default is us-east-1), which causes some confusion. Look at your App overiew on pusher.com and get the Cluster value from the 'Keys' section. Add this to your .env file as PUSHER_CLUSTER=

Create an AWS S3 bucket and grant it's permissions accordingly

  1. Log in to your AWS account and create an S3 bucket with a sensible name
  2. Now head to the AWS IAM (Identity and Access Management page)
  3. Click Users
  4. Add user (call it something sensible like octopub-development) and select Programmatic Access for Access Type.
  5. For permissions, select Attach existing policies directly - this will open a new tab in your browser.
  6. CLick create your own policy and give it a name, like octopub-dev-permissions, then for the policy document, use the following template, but add your own bucket name instead of <BUCKETNAME>.
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "AllowAdminAccessToBucketOnly",
           "Action": [
               "s3:*"
           ],
           "Effect": "Allow",
           "Resource": [
               "arn:aws:s3:::<BUCKETNAME>",
               "arn:aws:s3:::<BUCKETNAME>/*"
           ]
       }
   ]
}
  1. Click validate policy just to be sure you've not made a typo. Then confirm.
  2. Now back on the Set permissions page, select the policy you've just created in the table by selecting the checkbox. Then click Review then Create user.
  3. Now download the csv file containing the credentials and add the following to your .env file
AWS_ACCESS_KEY_ID=<YOURNEWUSERACCESSKEY>
AWS_SECRET_ACCESS_KEY=<YOURNEWUSERSECRET>
S3_BUCKET=<YOURNEWS3BUCKETNAME>

System dependencies and Configuration

Install mongo:
brew install mongo redis (if using brew)

make a data directory for mongo databases
sudo mkdir -p /data/db

change directory ownership so that mongodb can operate
sudo chown -R $USERNAME /data/

Development: Running the full application locally

Pre-requisites: AWS account, Pusher Account - these instructions assume you have these in place already.

Checkout the repository and run bundle in the checked out directory.

Database initialization

run mongo : mongod

run redis : redis-server

Services (job queues, cache servers, search engines, etc.)

to initialise the app run this in root directory of app

foreman start

Known issues & Troubleshooting

If you have trouble running bundle try these (if you see errors relating to openssl, eventmachine or therubyracer)

brew install openssl # if not installed
gem install eventmachine -v '1.0.7' -- --with-cppflags=-I/usr/local/opt/openssl/include
gem install libv8 -v '3.16.14.13' -- --with-system-v8
gem install therubyracer -- --with-v8-dir=/usr/local/opt/v8-315
https://github.com/shakacode/react-webpack-rails-tutorial/issues/266 ~ rubyracer with CSVlint

Tests

ensure phantomjs is installed, brew install phantomjs (or however you prefer)

To run the entire suite of rspec unit tests and cucumber features execute
bundle exec rake

alternatively execute each suite separately with

for unit tests execute bundle exec rspec

for Cucumber features execute bundle exec cucumber

Rake Tasks

rake csvs:clean_up
rake summary:generate
rake clean_up:dupes

Deployment

Deployment on Heroku