Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collections API: Update Celery to be Synchronous #2167

Merged
merged 6 commits into from
Aug 23, 2017

Commits on Aug 18, 2017

  1. Update AWS Credential Mapping

    Since the new geoprocessing service is run as the `mmw` user in
    the Worker VM, that user must have access to AWS credentials.
    Instead of mounting the developer's credentials into `/aws`, they
    are now mounted into the `mmw` user's home folder.
    
    Both `~/.aws/credentials` and `~/.aws/config` must be 644.
    rajadain committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    af6cddf View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2017

  1. Add sync methods for geoprocessing

    We add a task `run` and a helper method `geoprocess`. The `run`
    task converts the input into the desired format, and `geoprocess`
    communicates with the geoprocessing service and returns results.
    
    `run` is a combination of `start` and `finish`: it checks whether
    a result is cacheable and cached or not, and if so returns that.
    Otherwise it runs `geoprocess`.
    
    `geoprocess` is similar to `sjs_submit` in the sense that it is
    POSTing to an endpoint. Unlike `sjs_submit`, which gets back a job
    id, `geoprocess` receives the actual results and returns them.
    
    `run` is designed to replace `start` and `finish` tasks in Celery
    chains. So if a previous celery chain was:
    
        chain(geoprocessing.start.s(data),
              geoprocessing.finish.s(),
              mytasks.process_results.s())
    
    It will now be:
    
        chain(geoprocessing.run.s(data),
              mytasks.process_results.s())
    rajadain committed Aug 22, 2017
    Configuration menu
    Copy the full SHA
    22e8078 View commit details
    Browse the repository at this point in the history
  2. Replace start and finish with run

    We likely do not need to use `choose_worker` anymore, since each
    request is independent and can be run on any worker (in the right
    colored stack). However, this probably needs some more thought,
    and thus will be addressed in the separate issue #2117.
    rajadain committed Aug 22, 2017
    Configuration menu
    Copy the full SHA
    30327aa View commit details
    Browse the repository at this point in the history
  3. Remove unused tasks and methods

    These old async operations are no longer used.
    rajadain committed Aug 22, 2017
    Configuration menu
    Copy the full SHA
    88cf528 View commit details
    Browse the repository at this point in the history
  4. Use latest alpha of geoprocessing service

    This version includes RasterGroupedCount and RasterGroupedAverage
    operations, which make it sufficient for Analyze and TR-55 tasks.
    rajadain committed Aug 22, 2017
    Configuration menu
    Copy the full SHA
    df38e84 View commit details
    Browse the repository at this point in the history
  5. Update Ansible Java to latest with bugfix

    We need azavea/ansible-java#27 to solve
    AWS access issues with OpenJDK.
    rajadain committed Aug 22, 2017
    Configuration menu
    Copy the full SHA
    4e77070 View commit details
    Browse the repository at this point in the history