Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batch operations #61

Merged
merged 7 commits into from
Jun 3, 2020
Merged

Add batch operations #61

merged 7 commits into from
Jun 3, 2020

Conversation

alejgh
Copy link
Member

@alejgh alejgh commented Jun 3, 2020

Closes #41.

We have added a new way to perform operations more efficiently with Wikibase. The following features were added:

  • New function to optimise basic operations into batch operations.
    • Batch operations are considerably more efficient when there are many updates to execute for the same set of subjects.
    • All of these operations are executed in a single transaction, minimising the performance overhead of communicating with Wikibase.
  • Small refactor of the WikibaseAdapter class to minimise the calls to the wdi_core.WDItemEngine write method. We are making as many calls as possible to the update method instead, in order to reduce the writing operations to Wikibase and improve the overall performance.

This will unlock the start of #60, where we can illustrate with a visual approach the improvements that have been made.

@codecov
Copy link

codecov bot commented Jun 3, 2020

Codecov Report

Merging #61 into develop will decrease coverage by 0.06%.
The diff coverage is 99.11%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop      #61      +/-   ##
===========================================
- Coverage    98.08%   98.03%   -0.05%     
===========================================
  Files           15       15              
  Lines          728      760      +32     
===========================================
+ Hits           714      745      +31     
- Misses          14       15       +1     
Impacted Files Coverage Δ
hercules_sync/synchronization/algorithms.py 100.00% <ø> (ø)
hercules_sync/synchronization/operations.py 98.08% <96.78%> (-1.92%) ⬇️
hercules_sync/synchronization/__init__.py 100.00% <100.00%> (ø)
hercules_sync/triplestore/triple_info.py 99.39% <100.00%> (+0.02%) ⬆️
hercules_sync/triplestore/wikibase_adapter.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a583f55...f89aba8. Read the comment docs.

@alejgh alejgh merged commit bc8a821 into develop Jun 3, 2020
@alejgh alejgh deleted the batch_operations branch June 3, 2020 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant