Added partial update capability to sqlIngest #204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--Now able to specify page size, allowing larger pages if necessary
Bring dev up to date with upstream
--orm configuration files now in databaseOrm.py package --updateDatabase method of DataHandler now allows partial updates of records in database --runs roughly 90 seconds per 1000 records --around 5000 new records can be expected per day --fixed fetchSocrata method of DataHandler to correctly grab number of records specified in querySize parameter
Bringing up to date with main repo
--documented updateDatabase method --added a couple syntax/linting fixes to sqlIngest file
|
Oh snap...does this solve the upsert issue????????? |
Should do. It's slow, but it more or less works. Looks like sqlAlchemy doesn't have native upsert, so you have to jury rig a solution. There might be a more efficient way, but we should be able to grab the latest couple thousand records daily with this. |
|
Also oops on the linting. Looks like the linted file wasn't committed for some reason. |
sellnat77
approved these changes
Jan 26, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
--New functionality documented in /Documentation/ folder
--Now able to page arbitrary number of most recent records from a socrata store and integrate them into database
--Does not check whether records are changed, will update all records in most recent pull
--Currently runs somewhat slowly (~45 seconds/1000 updates) but this may be a limitation of sqlAlchemy