Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celerify HydroShare Export #2897

Merged
merged 9 commits into from Aug 15, 2018
Merged

Celerify HydroShare Export #2897

merged 9 commits into from Aug 15, 2018

Commits on Aug 9, 2018

  1. Use id instead of model in HydroShare Service

    Previously we would use a User Model to get an instance of
    a HydroShare Service Client for them. In this effort we're
    offloading some of this work to Celery Tasks, which do not
    support passing non-serializable Models as parameters. So,
    we will be forced to use simple data types.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    b9e6cde View commit details
    Browse the repository at this point in the history
  2. Add Celery tasks for creating, updating resource

    These mostly follow the create and update paths in the
    view with the addition of spawning their own copies of
    HydroShare Service, since that cannot be given to them
    across the Celery serialization boundary.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    820dddd View commit details
    Browse the repository at this point in the history
  3. Use new tasks instead of inline views

    Now, in the case of POST we run a Celery job instead of
    doing it all within a single request. This sends back a
    job_id that can be polled for. This polling will behave
    exactly as other such instances.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    b84a987 View commit details
    Browse the repository at this point in the history
  4. Adjust front-end to poll for results

    A HydroShareExportTaskModel is added to poll for results,
    set to poll every 6 seconds since this is relatively slow
    process. A task model gives us two promises: one for when
    the initial request is made, and another for when polling
    stops.
    
    We handle errors such as connectivity problems etc in the
    start promise failure. We handle response from HydroShare
    in the polling promise resolution. The exportToHydroShare
    method returns the two promises combined, to resolve when
    they are both ready. This isn't currently used, but keeps
    the API same as before when we returned the AJAX promise.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    a65ad60 View commit details
    Browse the repository at this point in the history
  5. Increase HydroShare Export timeout to 300s

    Since sending each file takes a few seconds, and when
    multiple scenarios are present in a project they have
    a large number of files, which can take a lot of time
    to send, we increase the timeout of both Celery tasks
    for creating and updating resources, as well as front
    end that will poll to a corresponding degree.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    5a5adaa View commit details
    Browse the repository at this point in the history
  6. Disable HydroShare Autosync

    Since now the HydroShare Export can potentially take
    quite a while, during which the user can keep making
    changes, with Autosync enabled that would simply add
    each change to the queue, leading to sitations where
    the app would constantly be exporting to HydroShare.
    
    Furthermore, because this export queue isn't visible
    to the user, they can navigate away, blocking export
    from completion.
    
    Adding a visible queue that is user-manipulatable is
    currently out of scope. Instead, our compromise here
    is to disable autosyncing completely, and relying on
    the user to export manually when they want.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    4bb58a2 View commit details
    Browse the repository at this point in the history
  7. Fix GMS file export

    When we switched to using job uuids for transferring
    MapShed results for Subbasin, we didn't update this
    HydroShare export functionality which had been broken.
    
    By switching to using the MapShed Job UUID, we enable
    GMS file exports to HydroShare again.
    rajadain committed Aug 9, 2018
    Copy the full SHA
    4c96a44 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2018

  1. Indicate changed project for HydroShare export

    Since we no longer autosync projects, and the user must
    be prompted to export explicitly, we add a message in the
    MultiShare Modal under the HydroShare section saying if
    the project has changed since last export.
    
    This message is only visible if the date of last export
    is less recent than the latest modified_at date of all
    the scenarios in the project. Thus, if any scenario is
    modified after an export, this message will be shown.
    
    To allow this, we now always update the client scenario
    model with the modified_at date from the server.
    rajadain committed Aug 15, 2018
    Copy the full SHA
    73f10d0 View commit details
    Browse the repository at this point in the history
  2. Only export changed scenarios

    Previously we always exported all scenarios for
    simplicity. Unfortunately, as we bump up against
    the limit of how many files can be exported before
    a timeout, especially when the record already exists
    and each export checks for a file, deletes it, and
    then exports it again, being judicious about which
    files we send ensures that we almost never time
    out.
    rajadain committed Aug 15, 2018
    Copy the full SHA
    20261d5 View commit details
    Browse the repository at this point in the history