api.ly.g0v.tw - ly api endpoint
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cookbooks/ly.g0v.tw
docker
scripts
src
.dockerignore
.gitignore
README.md
apiary.apib
app.ls
bill-analytics.ls
bill-details.ls
calendar-sitting.ls
dataflow.dia
dataflow.png
docgen.ls
g0v.json
ivod-montage.ls
lisproxy.psgi
live-firebase.ls
misq-proceeding.ls
msdl-live.ls
package.json
package.ls
populate-calendar.ls
populate-gazette.ls
populate-ttsbills.ls
populate-ttsinter.ls
populate-ttsmotions.ls
populate-video.ls
sitting-twitter.ls
twitter.json.sample
update-ivod.sh
ys-misq.ls

README.md

api.ly

gemnasium

api.ly.g0v.tw endpoint source and utility scripts

Development

  • Linux

    We recommend using docker for developing. Docker will install all the packages, configurations, database needed in a image (a file), not in your computer.

    • Workflow

      1. Install docker

        If you're using ubuntu / mint, please follow these steps: Remove sudo, Remove docker and local DNS server warnings

      2. Build base image

        $ ./docker/baseimage/scripts/build-image.sh
        
      3. Run database container (store database only, please see below)

      4. Run postgres container (please see below)

      5. Now you can

        • Run web server container
        • Run worker containers
        • Dig into database
        • Backup the database
        • Restore the database
    • Web server container (API endpoint)

      Build image

      $ ./docker/app/scripts/build-image.sh
      

      Open another terminal, and run:

      $ ./docker/app/run.sh
      

      Open you browser, see http://127.0.0.1:3000/collections/sittings

    • Woker containers

      Build image

      $ ./docker/workers/scripts/build-image.sh
      

      Open another terminal, and run:

      • Calendar worker

        $ ./docker/workers/calendar.sh
        
      • Sitting worker

        $ ./docker/workers/sitting.sh
        
      • Motion & Bill worker

        $ ./docker/workers/motion-and-bill.sh
        
      • Bill-details worker

        $ ./docker/workers/bill-details.sh
        
    • Postgres container

      Build image

      $ ./docker/postgres/scripts/build-image.sh
      

      Open another terminal, and run:

      $ ./docker/postgres/run.sh
      
      • Dig into database

        $ ./docker/postgres/scripts/psql.sh psql (9.3.4) Type "help" for help.

        ly=> \d List of relations Schema | Name | Type | Owner --------+-------------------+-------+------- public | amendments | table | ly public | bills | table | ly public | calendar | table | ly public | ivod | table | ly public | laws | table | ly public | motions | table | ly public | sittings | table | ly public | ttsbills | table | ly public | ttsinterpellation | table | ly public | ttsmotions | table | ly (10 rows)

    • Database container

      Build image

      $ ./docker/dbdata/scripts/build-image.sh
      

      Open another terminal, and run:

      $ ./docker/dbdata/run.sh
      
      • Backup the database

        $ ./docker/dbdata/scripts/backup.sh # save to backup/dbdata.tar
        
      • Restore the database

        $ ./docker/dbdata/scripts/restore.sh
        

        And restart postgres container

      Please backup the database first before stopping database container running.

    • Delete the images

      $ docker images
      REPOSITORY  TAG     IMAGE ID      CREATED         VIRTUAL SIZE
      lyapi-xxx   latest  569514b1aad4  42 minutes ago  1.439 GB
      $ docker rmi 569514b1aad4
      Deleted: 569514b1aad4...
      

      or simply:

      $ docker rmi $(doc images | grep lyapi- | awk '{ print $3 }')
      
  • Windows or Mac

    1. Install Virtualbox 4.3.x

    2. Install Vagrant 1.4.x

    3. Install berkshelf:

      $ sudo gem install berkshelf
      
    4. Install vagrant plugins:

      $ vagrant plugin install vagrant-berkshelf
      $ vagrant plugin install vagrant-cachier
      
    5. Run

      % cd cookbooks/ly.g0v.tw/
      % vagrant up
      

      You should now have localhost:6988 served by pgrest within the vagrant, try to access http://localhost:6988/v0/collections/sittings

Host

Besides Vagrant, of course you can run a api server in your host.

the server provides RESTFUL service by pgrest. pgrest rely on postgresql, so you should install postgresql and related components to your host.

For example, in Debian/Ubuntu

  • Create /etc/apt/sources.list.d/pgdg.list, add this line:

    Replace the codename with the actual distribution you are using.

    examples (distribution -> codename):

    Debian 7.0 -> wheezy

    Ubuntu 14.04 -> trusty

    deb http://apt.postgresql.org/pub/repos/apt/ codename-pgdg main
    
  • Import the repository key

    $ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
    $ sudo apt-get update
    
  • Install the packages.

    $ sudo apt-get install postgresql-9.3
    $ sudo apt-get install skytools3 skytools3-ticker postgresql-9.3-pgq3  # for pgq
    $ sudo apt-get install postgresql-9.3-plv8  # for plv8 extension
    
  • Preparation

    $ sudo su postgres
    $ export PLV8XDB=ly
    

init

% npm i

Then refer to the cookbook to initialize your postgresql.

Bootstrap with the initial dump file:

% createdb ly
% psql ly -c 'create extension plv8'
% lsc app.ls --db ly --boot
% curl -SsL 'https://www.dropbox.com/s/6j0zlzlubq8it5u/api.ly.bz2?dl=1' | bzcat |  psql ly -f -

run pgrest

$ lsc app.ls tcp://ly:password@localhost/ly
or
$ lsc app.ls tcp://ly:password@localhost:5433/ly    # if your postgresql is running on port 5433

pgrest will bind a local port to serve

data flow

fulltext from gazettes

(not automated yet)

ivod clip metadata

populated with update-ivod.sh

TTS data (WIP)

National Parliament Library provides a database called TTS. to work with it you'll need to have lisproxy.psgi up and running, and install phatomjs and casperjs to use scripts/tts.coffee

% sudo apt-get install cpanminus
% cpanm Plack::App::Proxy
% npm i -g bower
% npm i -g casperjs
% npm i -g phantomjs
% bower install jquery

% plackup lisproxy.psgi &

bill and motion metadata

populated with populate-ttsmotions.ls and populated-ttsbills.ls

interpellation

populated with populate-ttsinter.ls

API listing for api.ly

To regenerate api, just run docgen.s then commit apiary.apib, the document will hook to apiary automatically

cd ~/api.ly
lsc docgen.ls
git commit apiary.apib

License

MIT: http://g0v.mit-license.org