Improving Sync Responsiveness Between Kobo, GC and Superset

[nicopace]

This is a follow-up to a conversation we had with Rudo regarding how Windmill's data pulling from Kobo is not reactive to user actions.

As a result, users of the Guardian Connector may notice that submissions visible in Kobo are not immediately reflected in Superset dashboards (or any system querying the Postgres database) until Windmill runs its scheduled cron job.

One alternative is to retrieve data directly from Kobo using its REST services, which are well documented here:
https://support.kobotoolbox.org/rest_services.html

In practical terms, we could keep our current sync process but make it triggerable via the REST service for more timely updates.

[rudokemper]

This hasn't come up before as an issue or pain point for our users, since we typically set the Kobo script cron to be every hour which is pretty darn fast, and the interval could be made even shorter on demand. But nevertheless, I can imagine the use case. "I want to push a button to trigger the data fetch to happen on-demand."

However, that use case can also be met by just manually running the script in Windmill, once we start to expose the operator user role in Windmill to our users.

I agree that Kobo provides a more bespoke method for the need, and that it would therefore be good to use that in theory, but what value-add do we gain from making the effort to integrate with Kobo's REST services over what we already make available through Windmill? Are there other features / perks? (Asking because I don't know.)

[rudokemper]

My question is: why make the engineering effort to implement the change to begin with, if through Windmill we can already download data on-demand with our current script?

[nicopace]

The answer to that question would be because doing it this way would mean that the experience feels more integrated in comparison with the cron implementation. Of course, as no one has asked this, it's my impression and not a user-facing issue.

[rudokemper]

Unless you mean something different, I think the user experience in Windmill would be identical, and whether we fetch from API or REST service is a scripting implementation detail that the user would never see.

That is:

A user will encounter this screen, and hit "Run".

Even if we build a Windmill App for a nicer one-click button experience, it will still call a script under the hood. And it can use the existing API fetch script just the same.

[nicopace]

I guess the difference is in between accessing that screen to trigger it manually or to never have to trigger it at all by using the API, so the API does that trigger for them.

But I understand the point that there's already an interface for that.

I wonder if exposing the user to this user interface might imply that they not only go into this user interface, others that are more complex and by doing that we might want to prevent them from going through the hurdle.

[IamJeffG]

I wonder if there's a middle ground that is both of what you are saying: For people who really want the instant sync to Guardian Connector, the Kobo REST service is the mechanism that (through some sleight of hand by us) triggers the (same ol') Windmill script to run. (Windmill lets you trigger scripts via an endpoint)

Shorter term: I feel I'm hearing a bit of "we haven't shown anybody the current thing" vs "the current thing can't be right". Should we wait til we have Operator users of windmill, and then empirically watch if and how they use it? That will give us good data to re-evaluate this discussion.

I'm also hearing "current sync might not be quick enough" but I'm not hearing "people have said it's not quick enough". That would be another interesting data point to look at: what's the usual delta between posting Kobo responses and when someone actually goes to look at them in a real survey.

[nicopace]

I was looking at my notes, and I had a reference from a conversation with @rudokemper where he said something on the lines of

yesterday a***** wrote to rudo
When talking about this... but I don't know who A***** is... so can't share more... Rudo? :)

[rudokemper]

@nicopace since this is one of our public Github repos, I will respond to you on one of our private communication channels.

[nicopace]

was thinking if we have enough context from the user, that user could drive the conversation.