Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Background bulk add #262

Merged
merged 45 commits into from Apr 30, 2014
Merged

Background bulk add #262

merged 45 commits into from Apr 30, 2014

Commits on Apr 29, 2014

  1. Copy the full SHA
    72a06c8 View commit details
    Browse the repository at this point in the history
  2. Validate Mapping http_status is in array

    The constant was unused. Presumably it was once. I stringified the entries
    because we store them as strings in the database.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    13d666e View commit details
    Browse the repository at this point in the history
  3. Add validations to MappingsBatch

    These are mostly duplicates of things on Mapping, because MappingsBatch is a
    pseudo-mapping.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    862350b View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    91e4902 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    f6547ed View commit details
    Browse the repository at this point in the history
  6. Copy the full SHA
    4391083 View commit details
    Browse the repository at this point in the history
  7. Add Sidekiq

    Had to bump mlanett-redis-lock because it had a conflicting dependency on redis.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    6450d85 View commit details
    Browse the repository at this point in the history
  8. Copy the full SHA
    5e4d4c2 View commit details
    Browse the repository at this point in the history
  9. MappingsController uses background worker

    There is no longer any feedback to the user, which is lame. I'll add that back
    in next.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    e4b9688 View commit details
    Browse the repository at this point in the history
  10. Copy the full SHA
    6477912 View commit details
    Browse the repository at this point in the history
  11. Copy the full SHA
    0d19c9a View commit details
    Browse the repository at this point in the history
  12. Record state for each MappingsBatchEntry

    This will let us display progress to the user.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    9dbf400 View commit details
    Browse the repository at this point in the history
  13. Copy the full SHA
    20948ba View commit details
    Browse the repository at this point in the history
  14. Copy the full SHA
    dd3f01a View commit details
    Browse the repository at this point in the history
  15. Make spec deterministic

    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    82bd2a0 View commit details
    Browse the repository at this point in the history
  16. Copy the full SHA
    decf73c View commit details
    Browse the repository at this point in the history
  17. Clear old mappings batches

    We don't want ephemeral data hanging around forever.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    cb04e17 View commit details
    Browse the repository at this point in the history
  18. Disable Sidekiq retries

    Sidekiq will perform 25 retries over approximately 21 days using an exponential
    backoff algorithm. We probably don't want mappings being changed days afterward
    so let's never retry.
    
    This is fairly drastic - we could have 6 retries which would be attempted over
    6 minutes, but the MappingsBatch state field behaviour is incompatible with
    retries - it would mark them as failed after the first attempt.
    
    If we wanted to add retries back in, we'd need to change MappingsBatch's model
    of state to cope with it.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    0edfe20 View commit details
    Browse the repository at this point in the history
  19. Copy the full SHA
    36980d0 View commit details
    Browse the repository at this point in the history
  20. Copy the full SHA
    6140b81 View commit details
    Browse the repository at this point in the history
  21. Fix bug in detecting hosts for another site

    It wasn't deduplicating the hosts list, so when it compared with the (unique)
    list of hosts in the database, it gave the wrong answer.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    37ec109 View commit details
    Browse the repository at this point in the history
  22. Show number with delimiter

    Good UX, now that we have large batches of mappings.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    4d081ab View commit details
    Browse the repository at this point in the history
  23. Remove some redundant hidden fields

    These values are already in the database
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    b209462 View commit details
    Browse the repository at this point in the history
  24. Only show a preview of the batch

    Showing the whole of very large batches would cause very high view rendering
    times and hence timeout, plus it would have been unusable.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    7b019c3 View commit details
    Browse the repository at this point in the history
  25. Process small batches inline

    This makes much more sense when adding (possibly updating) a single mapping or
    a few mappings, which is definitely the normal case.
    
    This reinstates the modal confirmation dialogue for small batches.
    
    The specs for mappings_batch_outcome_presenter.rb were cribbed from the specs
    for BulkAdder.
    
    Reverted features changed earlier in this branch which run small batches to
    look for the modal, rather than the "x of y processed" flash message used for
    large batches.
    
    20 was chosen as the upper limit for a "small" batch because it should be easily
    processable within the 15 second timeout we have for admin apps, but is just
    above the average batch size according to Google Analytics.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    c2dc9f6 View commit details
    Browse the repository at this point in the history
  26. Create modal showing bulk add progress

    Cribbed from @fofr's prototype.
    
    Still needs:
    * some tidying up
    * the ability to dismiss it
    * not to be shown after it is dismissed/the page is refreshed
    
    * Spinner images from
    http://www.mytreedb.com/view_blog/a-33-high_quality_ajax_loader_images.h
    tml
    * Shows an updating count and progress bar
    * Doesn’t gracefully handle the adding/updating situation, needs
    different copy in that situation
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    16e3962 View commit details
    Browse the repository at this point in the history
  27. Move bulk progress script into a module

    * Switch from inline JS to a module
    * Pull URL from data attribute
    * Switch from setInterval to a timeout that gets set once the previous
    request is complete
    * Reduce scope of jQuery selectors to just the element we’re looking at
    * Create a spec
    
    Not yet handling errors, updating aria values or handling completion
    Paul Hayes authored and jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    3197210 View commit details
    Browse the repository at this point in the history
  28. Clean up background bulk add modal styles

    * Take out inline styles
    * Use default inherited styles where possible
    * Make some specific styles for this modal
    Paul Hayes authored and jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    7bd2846 View commit details
    Browse the repository at this point in the history
  29. Add an ok button to bulk add modal

    * Refactor modal CSS that deals with absolutely positioned close buttons
    * Make the modal-saved-mappings CSS re-usable, and remove the specific
    font-size: 80% rule on the glyphicon
    Paul Hayes authored and jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    3248246 View commit details
    Browse the repository at this point in the history
  30. Show a modal success state when all mappings added

    * When done matches total, stop making requests, show a success state
    and shut down the module
    * Use jasmine clock to avoid having to wait 1 second for each timeout
    test
    Paul Hayes authored and jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    6be5fe1 View commit details
    Browse the repository at this point in the history
  31. Copy the full SHA
    b0a7846 View commit details
    Browse the repository at this point in the history
  32. Fix whitespace

    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    7e37a4a View commit details
    Browse the repository at this point in the history
  33. Copy the full SHA
    a364cb1 View commit details
    Browse the repository at this point in the history
  34. Remove a redundant TODO

    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    f64a05e View commit details
    Browse the repository at this point in the history
  35. Remove redundant method

    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    1363441 View commit details
    Browse the repository at this point in the history
  36. Copy the full SHA
    f6101f0 View commit details
    Browse the repository at this point in the history
  37. Actually validate state on MappingsBatch

    The `validate` method is for validating with custom methods, not built-in or
    custom validator classes. Unfortunately, Rails doesn't complain that a method
    doesn't exist for the validation you've defined.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    c212ed1 View commit details
    Browse the repository at this point in the history
  38. Bulk add validation errors as per master

    The messages on master performed well in user research, whereas the messages
    generated by ActiveModel::Errors#full_messages did not. This change means that,
    for example, the #full_messages method returns messages like:
      "paths Enter at least one valid path"
    hence the change to `batch.errors.values.flatten(1)` in the view.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    71f21ca View commit details
    Browse the repository at this point in the history
  39. Copy the full SHA
    e10253c View commit details
    Browse the repository at this point in the history
  40. Remove redundant parameter to validate

    `validate` (as opposed to `validates`) doesn't take a field name argument, so
    doing so was redundant.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    0852e7c View commit details
    Browse the repository at this point in the history
  41. Find existing mappings on an indexed field

    This should be much faster, especially on large sites.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    d5ad0f2 View commit details
    Browse the repository at this point in the history
  42. Only show background modal once

    The flash message should continue to show on subsequent page loads to give
    feedback.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    84b6382 View commit details
    Browse the repository at this point in the history
  43. Copy the full SHA
    3ce45b7 View commit details
    Browse the repository at this point in the history
  44. Update the flash message with javascript

    In addition to displaying the modal (until dismissed), let's update the flash
    message with AJAX.
    
    A separate flash key was needed to separate it from the normal flash messages
    so that the required markup could be included without over-complicating the
    existing flash message code.
    jamiecobbett committed Apr 29, 2014
    Copy the full SHA
    a235b30 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2014

  1. Copy the full SHA
    b38ca2d View commit details
    Browse the repository at this point in the history