Remove CouchDB and django-sboard

This change removes CouchDB, django-sboard and everything that is related to
them.

For data scraping, CouchDB was replaces with MySQL via ManoseimasPipeline. Now
all scraped data will be added to tables specified in ManoseimasPipeline.models
attribute. Each model have `key` and `value` attributes that mimics CouchDB
documents.

If there are still places where CouchDB is uses, all those places have to be
replaced by `scrapy.models` models.

Website search was, previously based on CouchDB now replaced with Haystack
Search.

After refactoring I checked these places:

- Made sure, that all tests pass.

- Tried these commands and made sure they work:

      bin/scrapy crawl --loglevel DEBUG mps
      bin/scrapy crawl --loglevel DEBUG sittings
      bin/django recompute_stats

- Manually tried these parts of the page:

  - Front page

  - Website search

  - Widget

  - MP profile

  - Fraction profile

- Wrote more tests for mps_v2, widget and manoseimas apps.

- Tried to import production database locally and run migrations on it.

Everything listed above, more or less work. But this listed below does not
work:

- There is no user profile, this should be implemented with this #151

- Widget profile does not work, but this was already broken, before
  refactoring, when members of parliament was moved from CouchDB to MySQL.

- Compatibility test does not work, but this will be implemented here #155

.gitignore

@@ -27,6 +27,7 @@ pip-selfcheck.json
 /include
 /backup
 /node_modules
+/build/
 
 /settings.json
 /manoseimas/static/css/styles.css

HACKING.rst

@@ -16,14 +16,6 @@ Before starting, install system dependencies (this will require ``sudo``)::
 
     make ubuntu
 
-Run CouchDB using Docker::
-
-    docker run -d -p 5984:5984 --name couchdb klaemo/couchdb:1.6.1
-
-If you don't have Docker, you can install CouchDB using the manual way (see
-`Appendix: Manual CouchDB install`_) also you can install CouchDB using your package
-manager. It's up to you how you get CouchDB running.
-
 Build the project::
 
     make
@@ -32,10 +24,6 @@ Run migrations::
 
     bin/django migrate
 
-Sync CouchDB views for scrapy::
-
-    bin/initscrapy
-
 And run the project::
 
     make run
@@ -60,35 +48,6 @@ browser and navigate to::
     http://127.0.0.1:8000
 
 
-Appendix: Manual CouchDB install
-================================
-
-https://github.com/iriscouch/build-couchdb
-
-In short, CouchDB can be installed using these commands::
-
-    sudo apt-get install make gcc zlib1g-dev libssl-dev rake
-    git clone git://github.com/iriscouch/build-couchdb
-    cd build-couchdb
-    git submodule init
-    git submodule update
-    rake
-
-After building CouchDB, run it using this command::
-
-    ./build/bin/couchdb
-
-
-Production CouchDB access
-=========================
-
-To access CouchDB web UI run this command::
-
-    ssh -L 9000:localhost:5984 manoseimas.lt
-
-And visit http://localhost:9000/_utils/
-
-
 Manually running crawlers and housekeeping
 ==========================================
 
@@ -110,3 +69,25 @@ These commands are used to precompute and load various things::
 
 See the crontab rules in ``deployment/deploy.yml`` for the order and frequency
 of their execution.
+
+
+Running tests
+=============
+
+You can use this command for testing::
+
+  bin/django test \
+      -v 2 \
+      --settings=manoseimas.settings.testing \
+      --nocapture --nologcapture \
+      --all-modules --with-doctest --doctest-tests \
+      --with-coverage --cover-erase --cover-package manoseimas \
+      --noinput --failfast --keepdb \
+      manoseimas
+
+Note ``--keepdb`` flag, with this flag, database from previous test run will be
+reused. Usually this is a good thig, because tests will run much faster, but if
+database schema is changes, you need to create migration file and then recreate
+database::
+
+    $ mysql -uroot -e 'DROP DATABASE test_manoseimas; CREATE DATABASE manoseimas CHARSET=utf8;'

config/base.cfg

@@ -9,9 +9,6 @@ develop += .
 
 
 [sources]
-django-sboard = git https://github.com/ManoSeimas/django-sboard.git rev=fb8731f744f3ff14eab5a97253d69050ff6d7565
-couchdbkit = git https://github.com/sirex/couchdbkit.git branch=django-1.8
-restkit = git https://github.com/ManoSeimas/restkit.git rev=47da484371ed4029cebeb81f856860867f165705
 pyjade = git https://github.com/syrusakbary/pyjade.git rev=74e16e748eaf16957b6892476b49589787e23758
 
 

config/env/development.cfg

@@ -1,8 +1,5 @@
 [buildout]
 extends = ../base.cfg
-develop +=
-    ${buildout:parts-directory}/django-sboard
-    ${buildout:parts-directory}/couchdbkit
 parts += nose
 
 

config/versions.cfg

@@ -103,7 +103,7 @@ django-libsass = 0.4
 
 # Required by:
 # manoseimas-lt==0.1
-django-nose = 1.4.2
+django-nose = 1.4.3
 
 # Required by:
 # manoseimas-lt==0.1
@@ -265,3 +265,23 @@ Markdown = 2.6.6
 # Required by:
 # manoseimas-lt==0.2
 djangorestframework = 3.3.3
+
+# Added by buildout at 2016-06-02 15:30:18.741062
+WebOb = 1.6.1
+WebTest = 2.0.21
+waitress = 0.9.0
+
+# Required by:
+# WebTest==2.0.21
+beautifulsoup4 = 4.4.1
+
+# Required by:
+# manoseimas-lt==0.2
+django-webtest = 1.7.9
+
+# Added by buildout at 2016-06-03 17:09:16.528724
+Whoosh = 2.7.4
+
+# Required by:
+# manoseimas-lt==0.2
+django-haystack = 2.4.1

deployment/deploy.yml

@@ -162,20 +162,6 @@
       job: "cd {{ path }} && bin/scrapy crawl mps"
       user: "{{ name }}"
 
-  - cron:
-      name: Sync MPs
-      minute: 23
-      hour: 0
-      job: "cd {{ path }} && bin/django syncmps"
-      user: "{{ name }}"
-
-  - cron:
-      name: sittings crawl
-      minute: 45
-      hour: 0
-      job: "cd {{ path }} && bin/django syncsittings --scrape"
-      user: "{{ name }}"
-
   - cron:
       name: stenogram crawl
       minute: 5
@@ -190,13 +176,6 @@
       job: "cd {{ path }} && bin/scrapy crawl law_projects"
       user: "{{ name }}"
 
-  - cron:
-      name: sync positions
-      minute: 55
-      hour: 1
-      job: "cd {{ path }} && bin/django syncpositions"
-      user: "{{ name }}"
-
   - cron:
       name: recompute stats
       minute: 05

manoseimas/__init__.py

@@ -20,16 +20,10 @@
 
 This project is based on Django web framework and has these apps:
 
-mps
+mps_v2
     Members of Parliament.
 
-    This app mainly extends ``sboard.profiles`` functionality.
-
-solutions
-    Solutions are main object of this project and describes a solution to a
-    problem or to a set of problems.
-
-compat
+compatibility_test
     Compatibility checking app. This app helps to check if opinion on
     particular solutions or solution groups are compatible between different
     users and user groups.
@@ -38,7 +32,7 @@
     Scrapy scripts for crawling data from various sources. Main source of data
     currently is lrs.lt - web site of Lithuanian parliament.
 
-votings
-    MPs votings, automatically crawled from lrs.lt.
+widget
+    Embedable widget.
 
 """