Automatically run Rosie and update Jarbas #449
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This initial commit for this issue #447 suggests the usage of Ansible to create and delete a DigitalOcean droplet to ru these tasks. In between these steps we can run Roise and update Jarbas (but that's for next commits). Co-authored-by: Eduardo Cuducos <cuducos@ok.org.br> Co-authored-by: Filipe Cifali Stangler <cifali.filipe@gmail.com>
cuducos
force-pushed
the
cuducos-run-rosie-and-update-jarbas
branch
from
8160641
to
4ecdbe9
Compare
cuducos
force-pushed
the
cuducos-run-rosie-and-update-jarbas
branch
from
4ecdbe9
to
05f0cc3
Compare
16Gb was throwing MemoryError
|
Working on this issue I realized I oversimplified the Run the two update commands — the way I described before woulde nd up in data loss (namely reimbursement receipt texts, tweets and search vector). This step actually has to:
I'm working on a Django command to handle that using existing update commands ; ) |
cuducos
force-pushed
the
cuducos-run-rosie-and-update-jarbas
branch
from
e42016d
to
217457e
Compare
cuducos
changed the title
Beyond minor edits (fix tests, PEP8, simplify docs) this commit adds a new Django custom command to Jarbas and call it in the Ansible playbook. This command: * Backup Twitter data to re-link reimbursements later * Delete all data from Reimbursement model * Load all data from all reimbursements-YYYY.csv files * Load the suspicions.xz file * Reload receipt texts (download the CSV and import it) * Rebuild the search vector * Restores Twitter data
cuducos
force-pushed
the
cuducos-run-rosie-and-update-jarbas
branch
from
217457e
to
ebd05c8
Compare
CauanCabral
reviewed
Feb 1, 2019
Thanks @CauanCabral ;)
@chicocvenancio, well spotted. We must call it in the UPDATE Fixed in 1b1c515 |
As added to the README.md in this commit: 1. The `dopy` Python package Ansible depends on is [only available in Python 2](Wiredcraft/dopy#61) 2. We could use a fork, but we [would need to trust the fork owner](#449 (comment)) 3. Maintaining a fork is out of our scope at the moment However, Ansible is already [migrating for an alternative package](ansible/ansible#33984) and soon (before the end of life of Python 2) we will be able to update. Thanks, @willianpaixao for the heads up on item #2.
After feedback from the community (thanks @willianpaixao and @filipecifali) Ansible runs locally now (and not inside Docker anymore). This change also addresses the issue of creating a one-time only SSH key: now the process uses the host machine's SSH private key, and only requires the user to inform (as an environment variable) the name the public part of this key holds in DigitalOcean. It also uses Pipenv to be more explicit and restrict about the Python 2 requirement: by using it to run the commands it ensures the Python version specified in the Pipfile.
…-brasil/serenata-de-amor into cuducos-run-rosie-and-update-jarbas
Needed to run it locally, not in Docker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This work in progress PR kicks-offs the implementation for #447. More details of the implementation in the issue page.
Feel free to jump in this PR ; )
Gonna update this roadmap from time to time:
Create a DigitalOcean droplet982a729Installs Gitddbae12Install Docker431b9b0Install Docker Compose431b9b0Clone this repo44d6542Runs Rosie to generate the CSV282c7dd & 1147ab7Run Jarbas connecting to the production database2cc72c0Run the two update commands (reimbursements and suspicions)ebd05c8Delete the droplet982a729 & 0a5cc55