Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Migration Script #3

Conversation

mfraezz
Copy link

@mfraezz mfraezz commented Mar 1, 2016

Benchmarks

For a staging dump with ~60000 NodeLogs:

1) No optimization, just error handling:
 960.45s user 112.76s system 86% cpu 20:37.76 total
~4.5 hr on prod

2) Logging once per 1000 original nodelogs:
746.36s user 79.38s system 88% cpu 15:31.23 total
~3.4 hr on prod

3) No logging whatsoever:
737.77s user 77.76s system 89% cpu 15:12.22 total
~3.3 hr on prod

4) No logging, using pymongo for updates / inserts:
23.99s user 3.35s system 41% cpu 1:05.11 total
~6.66 min on prod

5) Logging all _id's and errors, using pymongo for updates / inserts:
71.48s user 8.50s system 52% cpu 2:31.12 total
~19.86 min on prod

Updates

#5 implemented.
Manually tested to verify pre-existing logs were working after this script was run.

pattisdr added a commit that referenced this pull request Mar 1, 2016
@pattisdr pattisdr merged commit c5516b9 into pattisdr:feature/eliminate_backrefs_between_nodes_and_nodelogs Mar 1, 2016
pattisdr pushed a commit that referenced this pull request May 5, 2017
…-next-url

[Fix] Further Updates For ORCiD [CAS-29]
pattisdr pushed a commit that referenced this pull request Oct 23, 2017
pattisdr pushed a commit that referenced this pull request Oct 23, 2017
* [MOD-29] Reviews Initial Commit

* [MOD-18] Add reviews to EXTERNAL_EMBER_APPS

* [MOD-20][MOD-21][Feature] Add reviews mixins (#21)

* [MOD-20][Feature] Add PreprintService.state field

* [MOD-21][Feature] Add ReviewProviderMixin

* Responding to review

* Use Enum

* Isolate transitions machine

* comments_public -> comments_private, for clarity

* Log model and read-only endpoints (#24)

* [MOD-12][MOD-17][MOD-39]Setup ember-osf-reviews app (OSF-side) (#22)

* [MOD-12] Setup ember-osf-reviews app
*Update docker setup
*Add osf-reviews campaign
*Add reviews route
*Add osf-reviews logo
*Update registration templates
*Update osf navbar
*Add mako landing page

* fix

* [MOD-22][MOD-24][MOD-25][MOD-48][Feature] Reviews API (#26)

* Add reviews fields to preprint/provider endpoints

* Add permissions...

* Cleaner permissions

* Apply permissions to ReviewLog endpoints

* Preprint permissions...

* Use reviews permissions on all preprint endpoints

* Filter preprint providers by reviews permission

* Add provider setup endpoint

* Allow node admins to see logs on their submissions

Assuming the provider settings allow it

* Hide log creators if comments are anonymous

* Add script to generate fake ReviewLogs

* Replace log state fields with action

* Creating review logs to trigger state transitions

* Allow resubmitting

* Fix admin

* Filter permissions correctly

* Responding to review

* Fix reviews migration

* Responding to review

* Fix stupid mistakes

* Cleaner PermissionHelper

* Fix provider log list view

* Add preprint list permissions tests

* Fix migrations after rebase

* Fix a couple things...

* No reviews workflow is None instead of 'none'

* Create review logs at /v2/reviews/review_logs/

* Add read-only title and contributors to preprints

* Hide preprints with private nodes from moderators

* Add date_last_transitioned field to preprints

* Expose live reload port for reviews

* Better permission names

* Tests for creating review logs, transitions

* Expose date_last_transitioned in API (#27)

* [MOD-61] Set reviews settings by PATCHing provider (#28)

* Nullable provider settings, set all at once (#29)

* Add related counts for reviewables (#30)

* Add related counts for reviewables

* Fixes

* CR

* [Fix] Hack around GuidQuerySet (#31)

* [Feature] More API tests and some cleanup (CenterForOpenScience#32)

* More reviews tests

* plac8 flake8

* Filter providers by multiple permissions with OR

* Tests and cleanup...

* More cleanup

* Responding to review

* Make travis happy

* Post-rebase cleanup

* Remove osf-reviews campaign

* Make travis happier...

* Add can_view_reviews to /users/me/ (#1)

* [Feature][MOD-73] Rename log to action (#2)

* ReviewLog => Action

* Move Action model to osf.models

* Fix up migrations

* Fix failing tests

* Fix up migrations again

* Make /actions/ useful in the browsable API

* Cleanup for consistency

* Add `action` relationship to users serializer

* Comment out reviews-specific dev docker stuff

* Responding to review

* Remove OSFReviews from navbar (#3)

* Fix provider detail endpoint

* Responding to review...

* Keep permissions in permissions classes

* Move scripts to management commands

* Revert "Keep permissions in permissions classes"

This reverts commit 51d8b4d.

* Reenable sorting tests

* Permissions in permission classes, mark 2

* Remove Reviews from the nav bar

* Add email notifications

* undo docker change

* replace review_log with action

* Fix style

* Update test_notifications

* Apply requested changes

* Add tests

* remove extra line

* Change wording

* Use preprint_word for providers

* Apply requested changes

* Remove print and fix space

* Apply requested changes

* Update test

* Apply requested changes

* Update test
pattisdr pushed a commit that referenced this pull request May 3, 2018
[#PLAT-509]Django-guardian for preprint permissions
pattisdr pushed a commit that referenced this pull request May 3, 2018
This reverts commit 4d9434f, reversing
changes made to 8acd0e8. 4d943f was
causing tests to fail and should be adjusted before being re-merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants