The repo contains the server & the client parts of the Open Pension project. Open Pension is a "Hasadna" project, that aimed to revealing the secrets behind the Israeli pension market.
We use docker, with docker compose, to manage all the services we use. There are three main services that help us to get the data properly:
- Fetcher - The fetcher service download for us all the source files from the pension companies.
- Processor - After we got the files we need convert them into JSON object which can be use for processing.
- Enrichment - The JSON objects we got from the fetcher used don't have fully qualified value. Each company can have various way to be represented - Teva, T.E.V.A, teva, tv25 etc. etc. etc. The service takes those variations and converts them to a unify texts.
The other services are peripheral services:
- BLOP - The data go into the
BLOPservice which is a
Flaskthat holds the data in a SQL tables.
- Gateway - The gateway service provides a single graphql endpoint which allow us to communicate with the other services.
- Client - A
reactapp that serves all the data.
- Backoffice - Gives us dashboard for managing other services:
- Content such as blogs, enrichment tables, user authentication and more
- Manage downloaded files, manually upload files for processing
- Watch logs from other services
- Fix uploaded files
- Kafka - give us the option to notify services of events that occurred in other services.
- Logs - Allow us to log events which occurred in service: a file has been processed, file downloaded.
- Install Docker (make sure you have docker compose).
docker-compose up -d.
- Make sure you have NodeJS
- Install Angular CLI
- move to client directory:
npm run build -- --watch
npm run lintto check for lint mistakes.
npm run testto execute the unit tests via Karma.
npm run e2eto execute the end-to-end tests via Protractor.
In order to execute any python commands you have to do it inside the service container
docker-compose exec [service-name] /bin/sh this will open the shell inside the container
pycodestyle --show-source --max-line-length=120 --exclude=pension/migrations --show-pep8 .to check for lint mistakes.
isort . --recursive --check-onlyto check for import mistakes.
python manage.py testto run the unit tetst.