DB handler updates #78
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
Acesonnall
commented
Oct 9, 2020
•
Acesonnall
commented
- DB handler support for latest scrapers
- converted latest scrapers passing phase 2 to work with db handler
- added function to set schema in mongodb
- added code to dump upload issues to a json file
- Iowa phase 2 complete and added to db
- Wisconsin phase 2 complete and added to db
…results later and documentation to wisconsin later as well
…gs to get it to work on your OS.
…gs to get it to work on your OS. For Arkansas scrapper, I sectioned everything out into individual functions to reduce complexity. Will document methods at a later time.
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
# Conflicts: # scrapers/michigan/michigan_scraper.py # scrapers/ohio/Ohio_Formatted.py # scrapers/ohio/ohio_scraper.py # scrapers/zip_county_mapping/zip_to_county.py
…es can be installed with ```pip install -r requirements.txt```. Updated some documentation.
* arizona scraper phase 2 complete. * california scraper complete Co-authored-by: Paulo Nascimento <pnascimento@uchicago.edu>
* north carolina done; florida init * Revert "north carolina done; florida init" This reverts commit 8d7cf96. * north carolina scraper * ohio scraper * maine scraper * Adding end of line character Adding new line to end of file to comply with PEP8 rules. * Adding end of line character Adding new line to end of file to comply with PEP8 rules. * florida scraper * initial michigan scraper * adding git ignore - completed michigan scraper * florida phase 2 * finished north carolina | maine in progress | slight tweak to michigan Co-authored-by: riya <riyaagar96@gmail.com> Co-authored-by: Omar Taylor <omtaylor@iastate.edu>
* removed misleading print msg from AZ | state initials -> full names for CA & NC * fixed misnamed field for california | added "X County Election Office"
* arizona scraper phase 2 complete. * california scraper complete * Minnesota scraper phase 2 complete Co-authored-by: Paulo Nascimento <pnascimento@uchicago.edu>
* arizona scraper phase 2 complete. * california scraper complete * Minnesota scraper phase 2 complete * texas phase 2 is done Co-authored-by: Paulo Nascimento <pnascimento@uchicago.edu>
…results later and documentation to wisconsin later as well
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
…results later and documentation to wisconsin later as well
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
…results later and documentation to wisconsin later as well
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
…results later and documentation to wisconsin later as well
…as specific by env vars. Also ran IDE and Black formatter and made other small updates to other files.
… they abide by formatting rules. Does not currently work for loading city election office data, but that's an easy fix. We didn't have any test data yet for that so I didn't want to get too ahead of myself.
# Conflicts: # definitions.py # scrapers/florida/florida_scraper.py # scrapers/north_carolina/north_carolina_scraper.py # scrapers/ptv_db_handler.py
…es server. Feel free to build off of this or just use for reference and delete things not needed.
…es server. Feel free to build off of this or just use for reference and delete things not needed.
…es server. Feel free to build off of this or just use for reference and delete things not needed. Also texas is passing validator
# Conflicts: # lib/scrapers/florida/florida_scraper.py # lib/scrapers/georgia/georgia_scraper.py # lib/scrapers/michigan/michigan_scraper.py # lib/scrapers/north_carolina/north_carolina_scraper.py # lib/scrapers/texas/texas_scraper.py
…rs that completed phase 2 to work with db handler
…rs that completed phase 2 to work with db handler
…lections. Updates to scrapers that completed phase 2 to work with db handler
Acesonnall
added this to Under Review
in Operation: Save the 2020 Election
via automation
This was
linked to
issues
Oct 9, 2020
pnasciment0
approved these changes
Oct 9, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.