Export Project to HydroShare

[rajadain]

Overview

This PR sets up backend and frontend components that allow the user to export a project to HydroShare. When a logged in user opens the MultiShare view in a saved project, they see the HydroShare Export switch. When this switch is turned on we first check to see if the user has authorized HydroShare. If not, we ask them to login and go through the OAuth authorization dance. With the authorization, we open a modal asking for the user to specify a title (defaulting to the project's name) and abstract, and optionally some keywords. When submitted, these values are combined with a new CSV export of each analysis type. Nothing from the modeling stage is currently included.

The above payload is sent to a new endpoint which verifies that the project exists and is owned by the user, then creates a resource in HydroShare with the given title, abstract, keywords, and files. It also adds a GeoJSON and Shapefile for area of interest of the project.

When this completes, the frontend updates to show a link to the project. Turning the switch off at this point opens a confirmation modal informing the user of the irreversibility of their action, pending which the resource is deleted from HydroShare.

This is a first draft, and there will be a number of follow up cards from this.

Connects #2544
Connects #2557

Demo

Notes

• We switch from using dev.hydroshare to beta.hydroshare on the suggestion of HydroShare developers. beta shares credentials with production, so our secrets file will have to be updated correspondingly.

• In case an analysis fails or is still running when the export is done, it will not be included in the export.

• The actual export takes quite some time: 10-15s. I'm not sure which step is the biggest bottleneck, I suspect it is the Shapefile generation, but haven't tested for sure. We can consider breaking this off into a Celery task with some persistent UI spinner to show this status outside of the Share modal.

• The CSVs generated for the Analyze tabs are markedly different from the ones retrieved through the UI. This is because the UI table CSVs work off rendered tables, however this is happening in the background and thus we work with raw model data. We can make these match in a future enhancement.

• A number of outstanding issues remain, which will be covered in later cards. A full list of these can be seen in HydroShare Export Planning #2507. Briefly, they are:

  • Group shapefiles in a folder for easy access
  • POST to /export/hydroshare?project=XX for an existing project should update that resource
  • Add UI for facilitating "export on save" so that every user action leads to a corresponding export
  • Analyze CSVs do not match those generated from the UI
  • Keywords should use bubbles in the UI
  • MapShed projects should export GMS files for every Scenario
  • Language and UI polish ("Unsychronize", error messages for edge cases)

  I'll be creating cards for these shortly.

Testing Instructions

• Check out this branch, reprovision app, run migrations, and bundle:

  $ MMW_HYDROSHARE_SECRET_KEY=*** vagrant reload app --provision
  $ ./scripts/manage.sh migrate
  $ ./scripts/bundle.sh --debug
  

• Go to https://beta.hydroshare.org/ and log in with credentials in the HydroShare OAuth MMW Beta LastPass site.

• Go to :8000/, log in, and create and save a project

• Go to the share menu, and enable HydroShare Export

• Authorize MMW

• In the Export modal, verify that the title matches the project name. Fill in an abstract, and optionally some keywords. Click "Export"

• Wait for the export to finish. When it does, there should be a link to the resource in HydroShare. Click it, and ensure it has everything advertised.

• Try downloading some files and ensure they work as expected.

• Go back to MMW, and switch off HydroShare. Once that completes, ensure that the resource has been deleted from HydroShare.

[rajadain]

@azavea-bot rebuild

[rajadain]

I can't seem to connect to the internal CI server, so will see if I can investigate from another computer.

[rajadain]

Ah I didn't add wellknown to the shrinkwrap. Adding it now.

[rajadain]

Updated credentials for staging and CI from dev.hydroshare to beta.hydroshare and using the new client secret. This is available in LastPass as well.

[jwalgran]

Works as described. Nicely done, and a nice, clear chain of commits made reviewing this substantial PR a breeze.

I left a few suggestions for improvement, but they are not blocking.

[jwalgran]

If the HydroShare URL changes, all of our urls will have to be updated with a data migration.

A SerializerMethodField may be an ideal alternative. Something like:

class HydroShareResourceSerializer(serializers.ModelSerializer):
    url = serializers.SerializerMethodField()

    class Meta:
        model = HydroShareResource

    def url(self, obj):
        return '{}resource/{}'.format(HYDROSHARE_BASE_URL, obj.resource),

[rajadain]

Great idea, will switch to that.

[rajadain]

Ended up going with a property on the model instead, which will allow its use elsewhere in the backend if we ever need it:

commit 339d70e59a44e87d4fde7ebdf053ff9488272df8
Author: Terence Tuhinanshu <ttuhinanshu@azavea.com>
Date:   Thu Dec 28 13:12:47 2017 -0500

    squash! Add HydroShareResource model, serializer, migration

diff --git a/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py b/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
index a8a18272..6705cef5 100644
--- a/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
+++ b/src/mmw/apps/modeling/migrations/0025_hydroshareresource.py
@@ -17,7 +17,6 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('resource', models.CharField(help_text='ID of Resource in HydroShare', max_length=63)),
                 ('title', models.CharField(help_text='Title of Resource in HydroShare', max_length=255)),
-                ('url', models.CharField(help_text='URL of Resource in HydroShare', max_length=1023)),
                 ('autosync', models.BooleanField(default=False, help_text='Whether to automatically push changes to HydroShare')),
                 ('exported_at', models.DateTimeField(help_text='Most recent export date')),
                 ('created_at', models.DateTimeField(auto_now_add=True)),
diff --git a/src/mmw/apps/modeling/models.py b/src/mmw/apps/modeling/models.py
index 039ab2db..d268edf5 100644
--- a/src/mmw/apps/modeling/models.py
+++ b/src/mmw/apps/modeling/models.py
@@ -5,6 +5,9 @@ from __future__ import division
 
 from django.contrib.gis.db import models
 from django.contrib.auth.models import User
+from django.conf import settings
+
+HYDROSHARE_BASE_URL = settings.HYDROSHARE['base_url']
 
 
 class Project(models.Model):
@@ -110,9 +113,6 @@ class HydroShareResource(models.Model):
     title = models.CharField(
         max_length=255,
         help_text='Title of Resource in HydroShare')
-    url = models.CharField(
-        max_length=1023,
-        help_text='URL of Resource in HydroShare')
     autosync = models.BooleanField(
         default=False,
         help_text='Whether to automatically push changes to HydroShare')
@@ -124,5 +124,10 @@ class HydroShareResource(models.Model):
     modified_at = models.DateTimeField(
         auto_now=True)
 
+    def _url(self):
+        return '{}resource/{}'.format(HYDROSHARE_BASE_URL, self.resource)
+
+    url = property(_url)
+
     def __unicode__(self):
         return '{} <{}>'.format(self.title, self.url)
diff --git a/src/mmw/apps/modeling/serializers.py b/src/mmw/apps/modeling/serializers.py
index 78d1e725..b73c18a3 100644
--- a/src/mmw/apps/modeling/serializers.py
+++ b/src/mmw/apps/modeling/serializers.py
@@ -67,6 +67,8 @@ class HydroShareResourceSerializer(serializers.ModelSerializer):
     class Meta:
         model = HydroShareResource
 
+    url = serializers.ReadOnlyField()
+
 
 class ScenarioSerializer(serializers.ModelSerializer):
 

[jwalgran]

Since the user does not receive any feedback letting them know that these fields are required the export button appears to do nothing if both fields are not filled in. A static "required" label near fields may be all we need.

[rajadain]

Added a small (optional) text to keywords which hopefully makes the other two more clearly required:

[rajadain]

I'm merging this since all the tests are passing on my local and in the interest of having something demo-able up on staging for RRP.