Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mjl index speedup #213

Merged
merged 23 commits into from
Oct 28, 2019
Merged

Mjl index speedup #213

merged 23 commits into from
Oct 28, 2019

Commits on Oct 17, 2019

  1. Use elasticsearch's parallel_bulk for indexing, add ELASTICSEARCH_DSL…

    …_PARALLEL default setting and parameters to management command.
    
    Use qs.iterator() for fetching data during reindex, as this is much more memory efficient and performant.
    Instead of finding out which methods to call to prepare fields, do that finagling once and cache it for subsequent model instance prepares.
    
    See issue django-es#154 for performance analysis and details.
    Martin J. Laubach committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    b4920c9 View commit details
    Browse the repository at this point in the history
  2. Move collection of prepare functions to __init__, where it's conceptu…

    …ally cleaner. Also shaves off a test per object.
    Martin J. Laubach committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    e5cd2df View commit details
    Browse the repository at this point in the history
  3. Minor cleanup: Move prepare cache to Document object instead of Model…

    …, as it's conceptually possible to have several indices on the same model.
    
    Also remove forced ordering that is a remnant of earlier code.
    Martin J. Laubach committed Oct 17, 2019
    Configuration menu
    Copy the full SHA
    861f704 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2019

  1. Configuration menu
    Copy the full SHA
    827800d View commit details
    Browse the repository at this point in the history
  2. Do not crash in init_prepare when no fields have been defined

    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    1773ed6 View commit details
    Browse the repository at this point in the history
  3. Crank up diff size to see what is going on

    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    acda7e7 View commit details
    Browse the repository at this point in the history
  4. Adapt test to changed call pattern

    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    8279746 View commit details
    Browse the repository at this point in the history
  5. Adapt tests to changed call patterns

    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    c4f230d View commit details
    Browse the repository at this point in the history
  6. Mark pagination test as expected failure for now.

    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    1dcc013 View commit details
    Browse the repository at this point in the history
  7. Define _prepared_fields as attribute in class so to_dict() won't pick…

    … it up as document field
    Martin J. Laubach committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    505406e View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2019

  1. remove debugging

    Martin J. Laubach committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    93f7d7c View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2019

  1. Add parameter no to do a count(*) before indexing, as for complex que…

    …rysets that might be expensive.
    Martin J. Laubach committed Oct 24, 2019
    Configuration menu
    Copy the full SHA
    c4a8a24 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2019

  1. Configuration menu
    Copy the full SHA
    2d2e996 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3b69009 View commit details
    Browse the repository at this point in the history
  3. Remove paginator test.

    Add tests for usage of init_prepare() and _prepared_fields.
    Add tests for correct calling of bulk/parallel_bulk.
    Martin J. Laubach committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    f613926 View commit details
    Browse the repository at this point in the history
  4. Make sure we compare w/ stable order

    Martin J. Laubach committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    51360c3 View commit details
    Browse the repository at this point in the history
  5. Adjust for different types for methods/partials in py2

    Martin J. Laubach committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    475e8a8 View commit details
    Browse the repository at this point in the history
  6. Correct es dependency (was conflicting with requirements.txt)

    Martin J. Laubach committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    45f62e7 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2019

  1. Pass queryset_pagination as chunk_size into parallel_bulk too.

    Martin J. Laubach committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    c933fb7 View commit details
    Browse the repository at this point in the history
  2. Add explanation why we use deque()

    Martin J. Laubach committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    96c3883 View commit details
    Browse the repository at this point in the history
  3. Correct typo in explanation of test

    Martin J. Laubach committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    9c138d6 View commit details
    Browse the repository at this point in the history
  4. Remove leftover instrumentation print

    Martin J. Laubach committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    9ec41a0 View commit details
    Browse the repository at this point in the history
  5. Better formatting to avoid backslash-continuation line

    Martin J. Laubach committed Oct 28, 2019
    Configuration menu
    Copy the full SHA
    5086396 View commit details
    Browse the repository at this point in the history