Return flow data results for merged datasets #1773

DavisRayM · 2020-01-29T10:10:47Z

Changes / Features implemented

Set uuid for MergedXForm on save
Return flow data results for merged datasets
Add test test_retrieve_responses_merged_dataset
Enable one to retrieve a FLOIP Package or response using the complete uuid4 string or its hex version
Add migration to set default uuid for XForms that are missing the field.
Return complete uuid4 strings on the floip viewset
Bump pyfloip version

Steps taken to verify this change does what is intended

Added test to ensure data is returned
Modified tests to ensure UUID is returned from the Merged XForm Endpoint

QA Checklist

Able to retrieve a MergedXForm and it's data from the FLOIP Endpoint.

Step 1: Retrieve MergedXForm UUID from the XForm endpoint /api/v1/forms/<form_id>
Step 2: Access the FLOIP Package http://localhost:8000/api/v1/flow-results/packages/<uuid>. Should return package information for the MergedXForm
Step 3: Access the FLOIP Responses http://localhost:8000/api/v1/flow-results/packages/<uuid>/responses. Should return all the answers to the form questions as JSON Objects
Things to consider: Forms with media, Forms with nested groups

uuid4 strings are used to identify packages(All ids returned by the endpoint should be uuid4). A uuid4 string can be identified by its 32 hex characters and 4 dashes

Closes #1643

DavisRayM · 2020-01-29T10:14:26Z

Side effects of implementing the PR as is

Need to find a way to set uuid for all pre-existing MergedXForms. Non breaking issue

The pre-existing MergedXForms defaulted theuuid to an empty string(Inherits the field from XForm). We may need to set the uuid fields for this feature/fix to work on merged datasets generated before this is implemented.

DavisRayM · 2020-01-29T10:17:57Z

onadata/apps/api/tests/viewsets/test_floip_viewset.py

+            response.data['attributes']['responses'])
+        # The transportation form(self.xform) contains 11 responses
+        # Assert that the responses are returned
+        self.assertEqual(len(response.data['attributes']['responses']), 11)


Chose to use the number of responses returned(11) for this test due to erratic time values returned in the responses.

DavisRayM · 2020-01-29T13:05:35Z

Side effects of implementing the PR as is
* Need to find a way to set `uuid` for all pre-existing MergedXForms. _Non breaking issue_
The pre-existing MergedXForms defaulted to an empty string(Inherits the field from XForm) as the uuid we may need to set the uuid fields for these xforms somehow.

Using a data migration to solve the missing uuid field value may be a bit painful for production servers since the XForm model it inherits from is usually quite bulky.

Other options we have is to set the values lazily either:

When a merged dataset is retrieved by the forms endpoint
When .save() is called on a dataset

Opinions on a favorable way to proceed ?

CC: @pld @ivermac

onadata/libs/serializers/merged_xform_serializer.py

ukanga · 2020-03-09T08:36:01Z

Using a data migration to solve the missing uuid field value may be a bit painful for production servers since the XForm model it inherits from is usually quite bulky.

Other options we have is to set the values lazily either:

When a merged dataset is retrieved by the forms endpoint

When .save() is called on a dataset

Opinions on a favorable way to proceed ?

I will recommend the data migration as well as the default value for the UUID field. This will be the data migration first and a second migration that sets the default value.

pld

can you please add a test for the generation of default IDs

DavisRayM · 2020-03-11T07:33:11Z

onadata/apps/logger/tests/models/test_xform.py

+        # Always have uuids.
+        # Assert that set_uuid is called with an object
+        # that already has a uuid.
+        self.assertEqual(set_uuid_mock.call_count, 1)


The DataDictionary model always sets uuid for XForms. Sadly, since the MergedXForm model inherits from the XForm model directly it doesn't have the uuid set...(And probably a few other custom processing steps ?)

I wonder if we should just modify the save method of the MergedXForm model to set the uuid ? I prefer having the uuid set directly from the parent class(XForm)

Yes, let's have a UUID for the merged form.

DavisRayM · 2020-03-11T07:34:04Z

can you please add a test for the generation of default IDs

I've added a test to test the generation of the default uuids in the latest commits. Please have a look.

DavisRayM · 2020-03-11T07:45:44Z

onadata/apps/logger/tests/models/test_xform.py

+        # Assert that the uuid is not empty and is
+        # 32 characters in length
+        self.assertNotEqual(uuid, '')
+        self.assertEqual(len(uuid), 32)


uuid4 is always 36 characters (32 hex digits + 4 dashes).

Decided to use the default way the uuid field is set for DataDictionary objects to not diverge from the norm; Generates uuids without the dashes(only hex)

ivermac

Code looks good to me but I'd like @pld or @ukanga to review as well

Add the 'uuid' field from the MergedXForm model to the tuple of fields to serialize in the MergedXFormSerializer. - Limit 'uuid' field to write only

Set the 'uuid' field on MergedXForms on save

faith-mutua · 2020-07-15T07:48:40Z

@DavisRayM Flow data results for merged datasets now have data.

ivermac

Generally looks good to me. I have one question though

ivermac · 2020-07-15T08:30:33Z

onadata/apps/logger/migrations/0061_auto_20200713_0814.py

+def generate_uuid_if_missing(apps, schema_editor):
+    """
+    Generate uuids for XForms without them
+    """
+    XForm = apps.get_model('logger', 'XForm')
+
+    for form in XForm.objects.filter(uuid=''):
+        form.uuid = get_uuid()
+        form.save()


Won't this be expensive when running migrations - even though it might be one time? Can we do this in the XForm's save function; check if that form has a uuid and set one if it doesn't exist.
I guess my worry is just the app going through all XForms checking which one doesn't have a uuid

May be not check on the XForm's instance save function but where the uuid is required

I'm not entirely sure if this query would be expensive to run(I'll try analyzing the query on a large database and get back to you). The XForm save function has a check to set the UUID when missing through the DataDictionary model.

The main reason this was included was because of MergedXForms since they're not created through the DataDictionary model. But, I've also modified the MergedXForm save to set it when missing...

I'm not sure at the moment about when MergedXForms are updated/saved again after creation. I'll update this once I research a bit more

Cool. I guess i'm fine if retrieving xforms without uuid won't be expensive

The query isn't expensive to run. The index we currently have on the UUID field speeds up the query quite a lot

ukanga · 2020-07-16T20:53:28Z

onadata/apps/logger/migrations/0061_auto_20200713_0814.py

+    """
+    XForm = apps.get_model('logger', 'XForm')
+
+    for form in XForm.objects.filter(uuid=''):


We tend to use xform instead of form as a variable of an XForm model object. Can we keep the same for consistency?

Yes, I've changed this to xform in the latest commits

Add a data migration that generates 'uuid' values for XForms that do not have a uuid

Return FLOIP results for merged datasets; Retrieve all instances attached to the forms that make up the MergedXForm

Bump pyfloip version to commit 3c980eb184069ae7c3c9136b18441978237cd41d

Support both uuid4 and uuid4 hex string on FLOIP Endpoint; Allow users to use the string to retrieve and publish FLOIP Responses & Results

Utilize uuid4 string when generating 'Location' header values for the FLOIP endpoint

Add ReadOnlyUUIDField custom serializer field to serializer UUID fields

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 57c4c86 to ea543e9 Compare January 29, 2020 10:15

DavisRayM commented Jan 29, 2020

View reviewed changes

DavisRayM changed the title ~~Fix issue where flow data results for Merged datasets would not be returned~~ Return flow data results for merged datasets Jan 29, 2020

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from ea543e9 to 10d560d Compare January 29, 2020 10:52

DavisRayM requested review from ivermac and pld January 29, 2020 13:01

DavisRayM mentioned this pull request Jan 30, 2020

ImproperlyConfigured: Could not resolve URL for hyperlinked relationship using view name "flow-results-detail". You may... #1637

Closed

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 10d560d to 614ab84 Compare February 5, 2020 10:02

DavisRayM mentioned this pull request Feb 7, 2020

Set a default uuid for the XForm uuid field #1779

Open

DavisRayM requested a review from ukanga March 9, 2020 07:43

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 614ab84 to 200aefe Compare March 9, 2020 07:46

ukanga reviewed Mar 9, 2020

View reviewed changes

onadata/libs/serializers/merged_xform_serializer.py Outdated Show resolved Hide resolved

lincmba previously approved these changes Mar 9, 2020

View reviewed changes

pld requested changes Mar 9, 2020

View reviewed changes

DavisRayM changed the title ~~Return flow data results for merged datasets~~ [WIP] Return flow data results for merged datasets Mar 9, 2020

DavisRayM dismissed lincmba’s stale review via a3f7fd2 March 11, 2020 07:30

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 20c3a0c to a3f7fd2 Compare March 11, 2020 07:30

DavisRayM commented Mar 11, 2020

View reviewed changes

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch 2 times, most recently from c6b2676 to 84c7033 Compare March 11, 2020 07:37

DavisRayM changed the title ~~[WIP] Return flow data results for merged datasets~~ Return flow data results for merged datasets Mar 11, 2020

DavisRayM commented Mar 11, 2020

View reviewed changes

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch 2 times, most recently from b97e3cd to a1ef783 Compare March 18, 2020 11:44

ivermac reviewed Mar 18, 2020

View reviewed changes

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from a1ef783 to e816e11 Compare March 18, 2020 12:21

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 33e1398 to dd535d0 Compare May 22, 2020 14:04

DavisRayM added 2 commits July 13, 2020 15:05

Add 'uuid' to the MergedXFormSerializers fields tuple

ee0baac

Add the 'uuid' field from the MergedXForm model to the tuple of fields to serialize in the MergedXFormSerializer. - Limit 'uuid' field to write only

Set MergedXForm uuid on save

583a1aa

Set the 'uuid' field on MergedXForms on save

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch 2 times, most recently from da3520e to 8820db3 Compare July 13, 2020 14:24

DavisRayM removed the QA+ PR passed QA testing label Jul 13, 2020

DavisRayM changed the title ~~[WIP] Return flow data results for merged datasets~~ Return flow data results for merged datasets Jul 14, 2020

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch 2 times, most recently from a1b7221 to 8a2a4a2 Compare July 15, 2020 06:36

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 8a2a4a2 to 6e10034 Compare July 15, 2020 07:52

faith-mutua added the QA+ PR passed QA testing label Jul 15, 2020

ivermac reviewed Jul 15, 2020

View reviewed changes

ivermac self-requested a review July 15, 2020 13:16

ivermac previously approved these changes Jul 15, 2020

View reviewed changes

ukanga requested changes Jul 16, 2020

View reviewed changes

DavisRayM added 8 commits July 17, 2020 09:29

Add migration that generates missing form uuids

77fee36

Add a data migration that generates 'uuid' values for XForms that do not have a uuid

Return FLOIP results for merged datasets

1fb2cae

Return FLOIP results for merged datasets; Retrieve all instances attached to the forms that make up the MergedXForm

Return complete uuid4 string for FLOIP Responses

5111ef2

Bump pyfloip version

d1a5573

Bump pyfloip version to commit 3c980eb184069ae7c3c9136b18441978237cd41d

Support both uuid4 and uuid4 hex strings on FLOIP endpoint

9ddee16

Support both uuid4 and uuid4 hex string on FLOIP Endpoint; Allow users to use the string to retrieve and publish FLOIP Responses & Results

Utilize uuid4 strings when generating 'Location' header value

d5f81ac

Utilize uuid4 string when generating 'Location' header values for the FLOIP endpoint

Add ReadOnlyUUIDField custom serializer field

b3c16d4

Add ReadOnlyUUIDField custom serializer field to serializer UUID fields

Return uuid4 strings instead of uuid4 hex strings

bcbf0b6

DavisRayM dismissed ivermac’s stale review via bcbf0b6 July 17, 2020 06:30

DavisRayM force-pushed the 1643-flow-date-result-merged-dataset branch from 6e10034 to bcbf0b6 Compare July 17, 2020 06:30

ukanga approved these changes Jul 17, 2020

View reviewed changes

DavisRayM merged commit 762ec1a into master Jul 17, 2020

DavisRayM deleted the 1643-flow-date-result-merged-dataset branch July 17, 2020 08:53

DavisRayM mentioned this pull request Jul 28, 2020

Tag release v2.3.6 #1856

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return flow data results for merged datasets #1773

Return flow data results for merged datasets #1773

DavisRayM commented Jan 29, 2020 •

edited

DavisRayM commented Jan 29, 2020 •

edited

DavisRayM Jan 29, 2020 •

edited

DavisRayM commented Jan 29, 2020 •

edited

Side effects of implementing the PR as is

ukanga commented Mar 9, 2020

pld left a comment

DavisRayM Mar 11, 2020 •

edited

ukanga May 6, 2020

DavisRayM commented Mar 11, 2020

DavisRayM Mar 11, 2020 •

edited

ivermac left a comment

faith-mutua commented Jul 15, 2020

ivermac left a comment

ivermac Jul 15, 2020

ivermac Jul 15, 2020

DavisRayM Jul 15, 2020 •

edited

ivermac Jul 15, 2020

DavisRayM Jul 15, 2020

ivermac Jul 15, 2020

ukanga Jul 16, 2020

DavisRayM Jul 17, 2020 •

edited

Return flow data results for merged datasets #1773

Return flow data results for merged datasets #1773

Conversation

DavisRayM commented Jan 29, 2020 • edited

Changes / Features implemented

Steps taken to verify this change does what is intended

QA Checklist

DavisRayM commented Jan 29, 2020 • edited

Side effects of implementing the PR as is

DavisRayM Jan 29, 2020 • edited

Choose a reason for hiding this comment

DavisRayM commented Jan 29, 2020 • edited

Side effects of implementing the PR as is

ukanga commented Mar 9, 2020

pld left a comment

Choose a reason for hiding this comment

DavisRayM Mar 11, 2020 • edited

Choose a reason for hiding this comment

ukanga May 6, 2020

Choose a reason for hiding this comment

DavisRayM commented Mar 11, 2020

DavisRayM Mar 11, 2020 • edited

Choose a reason for hiding this comment

ivermac left a comment

Choose a reason for hiding this comment

faith-mutua commented Jul 15, 2020

ivermac left a comment

Choose a reason for hiding this comment

ivermac Jul 15, 2020

Choose a reason for hiding this comment

ivermac Jul 15, 2020

Choose a reason for hiding this comment

DavisRayM Jul 15, 2020 • edited

Choose a reason for hiding this comment

ivermac Jul 15, 2020

Choose a reason for hiding this comment

DavisRayM Jul 15, 2020

Choose a reason for hiding this comment

ivermac Jul 15, 2020

Choose a reason for hiding this comment

ukanga Jul 16, 2020

Choose a reason for hiding this comment

DavisRayM Jul 17, 2020 • edited

Choose a reason for hiding this comment

DavisRayM commented Jan 29, 2020 •

edited

DavisRayM commented Jan 29, 2020 •

edited

DavisRayM Jan 29, 2020 •

edited

DavisRayM commented Jan 29, 2020 •

edited

DavisRayM Mar 11, 2020 •

edited

DavisRayM Mar 11, 2020 •

edited

DavisRayM Jul 15, 2020 •

edited

DavisRayM Jul 17, 2020 •

edited