MQTT Pushes for Form Updates

[jasonrogena]

Add support for pushing form updates and free form messages to MQTT.

Form Schema Updates

These updates correspond to changes in XForms forms within Onadata. Form actions that qualify as schema updates include:

1. Publishing a form under an Ona account.
2. Unpublishing/deleting a form (includes moving a form away from an Ona account).
3. Changing the form name or description.
4. Updating a form's structure by uploading an updated XLSForm.
5. Adding a media file (including linked data-sets) to the form.
6. Removing a form's media file (including linked data-sets).
7. Updating a form's media file (including linked data-sets).

An MQTT payload for this type of update should look like:

{
  "uuid": "cf339f09fbfc38766de5cc01ddc5e6c4",
  "time": "2017-01-01T19:20:30.445+01:00",
  "payload": {
    "formID": "form_2016IRS_HH",
    "name": "2016 HH Submission Form",
    "majorMinorVersion": "",
    "version": "",
    "hash": "md5:74bd6634aa4c8c510b2c1018d2b55e06",
    "descriptionText": "",
    "downloadUrl": "https://odk.ona.io/jrogena/forms/153798/form.xml",
    "manifestUrl": "https://odk.ona.io/jrogena/xformsManifest/153798"
  }
}

Payloads can be pushed to one of the following topics, depending on the action:

• onadata/users/jrogena/forms/153798/schema/published
• onadata/users/jrogena/forms/153798/schema/updated
• onadata/users/jrogena/forms/153798/schema/unpublished

The topics follow the following syntax:

onadata/users/[Onadata user or org]/forms/[Onadata form primary key]/schema/[action]

Topics can contain up to 32,767 UTF-8 (no other character encoding supported) characters. Note the use of verbs (in past tense) in the lowest bit of the topics. My way of showing that final topic level represent an action (and not an object). Higher topic levels should represent objects.

Free Form Message Updates

These updates correspond to actions on user-defined messages published on Ona. Free form messages will only be tied to forms (users should be able to send free-form messages relating to forms). I, however, anticipate that in future this type of message could be tied to other Ona object types (orgs, users, projects e.t.c.).

An MQTT payload for this type of update should look like:

{
  "uuid": "ddd39f09fbfc38766de5cc01ddc5e6c4",
  "time": "2017-01-01T19:20:30.445+01:00",
  "payload": {
    "id": "dfestt3sbtdsfd24dfdsfehssete32fed",
    "author": {
      "username": "jrogena",
      "real_name": "Jason Rogena"
    },
    "context": {
     "type": "form",
     "metadata": {
        "name": "2016 HH Submission Form",
        "formId": "form_2016IRS_HH"
      }
    },
    "message": "Quick brown fox does what?"
  }
}

The value for the payload.message JSON key can contain up to 65,536 Unicode characters.

Payloads can be pushed to one of the following topics, depending on the action:

• onadata/users/jrogena/forms/153798/messages/ddd39f09fbfc38766de5cc01ddc5e6c4/published
• onadata/users/jrogena/forms/153798/messages/ddd39f09fbfc38766de5cc01ddc5e6c4/updated
• onadata/users/jrogena/forms/153798/messages/ddd39f09fbfc38766de5cc01ddc5e6c4/unpublished

The topics follow the following syntax:

onadata/users/[Onadata user or org]/forms/[Onadata form primary key]/messages/[message ID]/[action]

Topics can contain up to 32,767 UTF-8 (no other character encoding supported) characters. Note the use of verbs (in past tense) in the lowest bit of the topics. My way of showing that final topic level represent an action (and not an object). Higher topic levels should represent objects.

[pld]

Blocking questions:

• Under what conditions would we want to push an update to MQTT? All, controlled, parameterized? If not all, how so?
• How should we configure the MQTT server that the API pushes updates to? Are all updates pushed to the same MQTT server, would the server ever be different depending on the user/form/server/cluster? Will we ever need to hot-swap the MQTT server without a server restart?
• How is the message_id set?
• What content builds the hash?
• How/should we fill the empty fields majorMinorVersion, version, and descriptionText?

Non-blocking questions:

• Do we want to use MQTT for other messaging/queueing now or in the future?
• Is there a canonical python library we want to use for this?

CC @ukanga @denniswambua any more questions scenarios you can think of?

[jasonrogena]

@pld Here you go:
"Under what conditions would we want to push an update to MQTT? All, controlled, parameterized? If not all, how so?"

• My opinion is to push this message when a form is created or updated (I've updated my comment on what qualifies as an update). I'd also suggest we push the same message under the onadata/users/[publishing user]/forms topic (can be done later).
• UPDATE: I've updated the topics to include the user (or org) owning the form. It should now be possible to subscribe to onadata/users/[username]/* and receive push notifications on newly published forms on that account. No need to publish form updates in two topics (onadata/forms/[form id] and onadata/users/[publishing user]/forms)

"How should we configure the MQTT server that the API pushes updates to? Are all updates pushed to the same MQTT server, would the server ever be different depending on the user/form/server/cluster? Will we ever need to hot-swap the MQTT server without a server restart?"

• All updates can be pushed to the same DNS name (push.ona.io is what I set in the Playbooks). That can directly point to one MQTT broker, that should be able to handle all the pushes, or a load balancer hooked to several brokers.
• A single broker should be able to handle different messages (sent to different client types; Collect, Zebra e.t.c). This will largely depend on how well we define our [topic hierarchies].(http://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices).
• There's no denying that, in the future, we might need to hot-swap the Mosquitto broker while everything else is live. That however shouldn't be a problem if we have at least two running behind a load balancer.

"How is the message_id set"

• That can be anything that uniquely identifies the message (UUID or hash of the rest of the message should suffice). UPDATE: I've renamed this variable to 'uuid' which I think is more descriptive.

"What content builds the hash?
How/should we fill the empty fields majorMinorVersion, version, and descriptionText?"

• All the fields under the payload key are JavaRosa fields that are also available for each xform in the /formList endpoint (https://odk.ona.io/formList). I noticed that some of these fields were blank on the /formList endpoint but chose to keep them there in compliance with the standard.

"Do we want to use MQTT for other messaging/queueing now or in the future?"

• The MQTT protocol excels at message pushing, but we'd probably want to stick with AMQP (what RabbitMQ uses) for message queues, since it's more geared towards message queueing.

"Is there a canonical python library we want to use for this?"

• The Eclipse project (they also maintain Mosquitto) has a Python client that might be a good pick.

[jasonrogena]

The push should also fire when linked datasets are updated. I've updated my initial comment with this.

[cafootitt]

I was discussing this quickly with Jason, and it would be good to add the ability to notify the user to a form update if any linked datasets (e.g. pull csv, linked dataset, media, etc.) was changed.

[lincmba]

There has been work already done as regards publishing messages to MQTT here #1288. The root topics there are:

• /onadata/users/#
• /onadata/forms/#
• /onadata/projects /#
  There has been further development here Use MQTT to send updates upon submission review update #1791 (comment) specifically on the /onadata/xform/# topic.

However, the topics' definition in the above work doesn't follow the hierarchy of objects in Onadata. i.e, Organisations have Projects, and Projects have XForms, and XForms have Submission.

Following discussions with @ukanga , the hierarchy of the topic definitions needs to be made in such a way that allows more flexibility. An example topic would be:
/organization/[org_id]/project/[proj_id]/xform/[xform_id]/#

In such, there's flexibility in narrowing down or widening what to listen to by using a single topic root.

[ukanga]

Is there a reason we may want namespaces for topics? Like sharing the MQTT service between different implementations and services? Such that we do

/[namespace]/organization/[org_id]/project/[proj_id]/xform/[xform_id]/#
i.e
/onadata/organization/[org_id]/project/[proj_id]/xform/[xform_id]/#
/opensrp/[org_id]/project/[proj_id]/xform/[xform_id]/#

[lincmba]

Is there a reason we may want namespaces for topics? Like sharing the MQTT service between different implementations and services? Such that we do

Yes. The service might be shared and it would be great to have.

[ukanga]

I think with the namespace, your proposal will be good to go. I would expect we will still maintain the previous topics, perhaps have them on a depreciation path.