📱Kobotoolbox API and Trigger nodes #2510

Yann-J · 2021-12-01T22:08:55Z

KoboToolbox is a popular open-source field survey / data collection tool largely used by humanitarian organizations.

This PR has support for most API calls (no trigger...): fetching form definitions, submissions, and webhooks.

All tested, code linted, all should be pretty polished...

CLAassistant · 2021-12-01T22:08:59Z

All committers have signed the CLA.

Yann-J · 2021-12-03T19:50:38Z

Hello there!

Sorry for the multiple commits, but I kept thinking of more features to add, including light reformatting of the form submission data, and a trigger node to receive submission notifications.

The PR is now finished as far as I can tell - tested, linted and snyk check finally passed!

Yann-J · 2021-12-14T13:15:45Z

Hey @janober wondering if there's anything I can do to help with the review? I've been happily adding some polish over the last few days but I think it's basically finished...

Let me know if it looks good!

Yann-J · 2021-12-15T06:20:29Z

X-ref the corresponding docs PR: n8n-io/n8n-docs#678

Yann-J · 2022-01-12T09:46:30Z

Hello there @ivov @janober ... Let me know if there's anything I can do for a speedy review... we are eagerly waiting for this node to be available in our setup...

csuermann · 2022-01-14T08:41:24Z

Hi @Yann-J, sorry for the delay. We have a growing number of nodes to review but this one should be coming up quite soon.

RicardoE105

@Yann-J got reviewed. Let me know if you have questions about the comments.

packages/nodes-base/credentials/KoboToolboxApi.credentials.ts

packages/nodes-base/nodes/KoboToolbox/GenericFunctions.ts

packages/nodes-base/nodes/KoboToolbox/KoboToolbox.node.ts

packages/nodes-base/credentials/KoboToolboxApi.credentials.ts

packages/nodes-base/nodes/KoboToolbox/KoboToolbox.node.ts

packages/nodes-base/nodes/KoboToolbox/KoboToolboxTrigger.node.ts

packages/nodes-base/nodes/KoboToolbox/descriptions/GeneralOptions.ts

packages/nodes-base/nodes/KoboToolbox/test.rest

Yann-J · 2022-01-18T16:16:53Z

Thanks a lot @RicardoE105 for the thorough review!

I've implemented most of the suggested changes!

The only exceptions are:

I'm reluctant to rename the submission query operation to "Get All", because I expect it would be quite rare to really actually get all records - this will often be a lot. I suspect most users will use a query filter, so query feels more appropriate here...
I have not implemented support to dynamically load the form ids from the UI... I understand it would be convenient, but I think it can be done later as I lack time to do this properly (there's a little bit of work since the formList API is actually served by a different domain from the main one, and is at risk of deprecation), and I'm a little concerned that this can be a lot for heavy users (many organizations would share all forms with all users...).

Somehow, credentials testing always seems to work for me, on all the servers I've tested this on... I couldn't figure out anything problematic...

Yann-J · 2022-01-21T15:08:25Z

OK @RicardoE105 so I was curious about this possibility of looking up the form ID, and gave it a shot - it does make the experience smoother for sure!

Added lookup in both API and Trigger nodes...

Thanks for the tip!

RicardoE105 · 2022-01-24T00:51:10Z

Great that it was helpful.

I will review it again when I have time.

RicardoE105 · 2022-01-24T21:19:07Z

@Yann-J ok gave it a try and found a couple more issues. Can you please address those? Then I will do more testing, fix whatever is left (it should be none or really small stuff), and move internally in the node pipeline. I tried to address them myself since it should be very quick but did not find the Rest API docs. Let me know if you have any questions about my comments.

RicardoE105 · 2022-01-24T21:24:48Z

Also, if you know where the docs about the API are, let me know. Thanks.

RicardoE105 · 2022-01-24T22:11:11Z

Also, I just noticed that the name of the tool is KoBoToolbox, but you used it in the node KoboToolbox. This needs to be updated in the node and in the credentials.

Yann-J · 2022-01-25T07:05:15Z

Indeed @RicardoE105 , I believe the new spelling is due to a new branding... I've updated it everywhere.

Regarding the API docs, AFAIK they're not publicly available, but you can browse the live API endpoints if you are logged in to a running instance, e.g. at https://kf.kobotoolbox.org/api/v2/assets/ - the HTML views include the docs.

Thanks for everything!

RicardoE105 · 2022-02-02T15:23:00Z

@Yann-J any update on this?

Yann-J · 2022-02-02T15:35:56Z

Hi @RicardoE105 yes, I believe my latest commit has addressed the branding change.

Is there anything else I owe you?

RicardoE105 · 2022-02-02T20:23:18Z

@Yann-J Ah looks like I never submitted the changes. My bad.

packages/nodes-base/nodes/KoboToolbox/descriptions/FormQueryOptions.ts

RicardoE105 · 2022-01-24T20:45:30Z

packages/nodes-base/nodes/KoboToolbox/descriptions/GeneralOptions.ts

+		name: 'assetUid',
+		type: 'options',
+		typeOptions: {
+			loadOptionsMethod: 'loadSurveys',


Do not load the forms for the operation form:get.

Why is that?

packages/nodes-base/nodes/KoboToolbox/descriptions/HookOptions.ts

RicardoE105 · 2022-01-24T21:03:52Z

packages/nodes-base/nodes/KoboToolbox/descriptions/SubmissionQueryOptions.ts

+		name: 'start',
+		type: 'number',
+		displayOptions: {
+			show: {


In the operation submission:query the pagination is missing.

The query collection should be rename filters, we need to remove the sort and fields parameter. Those two parameters should be moved under options collection.

Get rid of the parameter start since the pagination should be done for the user.

Rename collections' button name. By default it says "choose option to add" but it should be "Add Option". You can this name by setting the property placeholder to the collection.

I must say I'm not really convinced about this split menu structure, keeping filters, sort, and fields under the same Query Options collection make more sense to me, as they're the 3 typical options you'd expect from a query operation.

Besides they all have the same displayOptions (when I try to move some under the generic options collection, I'm getting a Could not resolve parameter depenencies. Max iterations reached! error...)...

Also I've kept the start parameter because I'm certain that some workflows will want to implement their own pagination (somehow... it might be tough... but some forms will have millions of submissions, we can't load them all in memory...).

packages/nodes-base/nodes/KoboToolbox/KoboToolbox.node.ts

Yann-J · 2022-02-03T09:54:35Z

OK yes you're right @RicardoE105 , I should be handling the pagination, I will work on it... However I would still think keeping the start parameter is useful, because I think there would be cases where the user might want to handle pagination themselves - e.g. to avoid fetching 500k records (which is the case on some of our forms), and process them by batch.
It won't be the case for forms and hooks presumably, but it could be for submissions and hook logs...

Yann-J · 2022-02-03T10:50:23Z

Hmm, on the other hand, the node will only return the result list anyway, and not the pagination links, so it might be hard to implement pagination outside the node anyway...

Yann-J · 2022-02-03T13:34:30Z

OK @RicardoE105 , I believe this last commit has addressed most comments... I don't fully agree with some of the menu structure changes, so I've kept them as is, but pagination is alive and well on all list-based operations.

RicardoE105 · 2022-02-07T23:05:27Z

@Yann-J could not test the hook resource. When I create a hook on a form and then try to retrieve all the hooks in that form using hook:getAll I always get empty. Any idea what I might be doing wrong?

Yann-J · 2022-02-08T05:41:16Z

Hmm... it seems to be working for me. Could it be a permission issue, e.g. if you created the hook with a different user than the one used by n8n?

I've pushed a commit that fixed a linting error... maybe that was causing a build issue or something.

Yann-J · 2022-02-09T14:26:38Z

FYI, just committed one last change to lookup hook IDs

RicardoE105 · 2022-02-15T21:07:47Z

@Yann-J quick update. I'm waiting for a PR extending the method httpRequest to be merged so move forward with this PR. That functionality would allow following the redirects using the httpRequest method instead of having import axios to do it.

Yann-J · 2022-02-16T06:51:10Z

Ok thanks Ricardo... Hope this can happen soon since we're really eager to use this one...

Yann-J · 2022-03-03T05:09:06Z

Hey there @RicardoE105 wondering if there's any update here... we've been really waiting for this node...

RicardoE105 · 2022-03-07T16:43:15Z

@Yann-J sorry for the late response. Somehow missed the notification. The PR I was waiting for got merged. So I'm gonna move it forward this week. Hopefully will be available in the next release.

* First version * Added hooks * Added Credentials test * Add support for downloading attachments * Slight restructure of downloaded binaries * Added Trigger node * Some linting * Reverting package-lock changes * Minor GeoJSON parsing fixes * KoboToolbox: improve GeoJSON format * Kobo: Support for get/set validation status * Remove some logs * [kobo] Fix default attachment options * Proper debug logging * Support for hook log status filter * Kobo: Review fixes * [kobo]: Add Get All Forms + lookup Form ID * [kobo] Lookup Form ID in Trigger node * [kobo] Update branded spelling * [kobo] Support pagination * ⚡ fix linting issue * ⚡ Improvements to #2510 * ⚡ Download files using n8n helper * ⚡ Improvements * ⚡ Improvements * 🐛 Fix filenames * ⚡ Fix some issues Co-authored-by: Yann Jouanique <yann.jouanique@oneacrefund.org> Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>

janober · 2022-03-20T21:19:38Z

Got merged with #2765 and released with n8n@0.169.0

* ✨ Make it possible to download binary data * ⚡ Fix lint issues and add support for filesystem mode * ⚡ Design adjustment 📜 Change to the Sustainable Use License n8n-io#2932 🚨 Temporarily skip some regularly failing tests (n8n-io#3002) feat: Add support for reading ids from file with executeBatch command (n8n-io#3008) feat(Mattermost Node): Add support for Channel Search (n8n-io#2687) * Squashed commit of the following: commit 9f76bdca9b4af4fd3ee429d1c381c3ed5529434c Author: Matt Walther <matt@mashio.net> Date: Sun Jan 16 16:40:34 2022 -0600 Mattermost Channel Search * Add more boilerplate so Search action renders * Changed order of search to make the operations alphabetical * :zap: Add pagination bug(EmailReadImap Node): Improve error handling (n8n-io#2991) * Fix: EmailReadImap unhandled promise rejection Related to n8n-io#2091 (but only partially) See n8n-io#2091 (comment) * Send errors from email read imap to logger feat(HTTP Request Node): Allow Delete requests with body (n8n-io#2900) delete request with body parameters feat(KoBoToolbox Node): Add KoBoToolbox Regular and Trigger Node (n8n-io#2765) * First version * Added hooks * Added Credentials test * Add support for downloading attachments * Slight restructure of downloaded binaries * Added Trigger node * Some linting * Reverting package-lock changes * Minor GeoJSON parsing fixes * KoboToolbox: improve GeoJSON format * Kobo: Support for get/set validation status * Remove some logs * [kobo] Fix default attachment options * Proper debug logging * Support for hook log status filter * Kobo: Review fixes * [kobo]: Add Get All Forms + lookup Form ID * [kobo] Lookup Form ID in Trigger node * [kobo] Update branded spelling * [kobo] Support pagination * ⚡ fix linting issue * ⚡ Improvements to n8n-io#2510 * ⚡ Download files using n8n helper * ⚡ Improvements * ⚡ Improvements * 🐛 Fix filenames * ⚡ Fix some issues feat(Linear Node): Add Linear Node (n8n-io#2971) * ✨ Linear node * ⚡ Improvements fix(GitHub Node): Fix credential tests and File > List operation (n8n-io#2999) * Fixed credential test failing * Fixed File list operation not working fix(Telegram Node): Fix sending binary data when disable notification is set (n8n-io#2990) feat(Mailjet Node): Add credential tests and support for sandbox, JSON parameters & variables (n8n-io#2987) * Add Variables JSON to Mailjet Batch send * ⚡ Improvements * ⚡ Add credential verification * ⚡ Small improvement ⬆️ Update package-lock.json file 🔖 Release n8n-workflow@0.92.0 ⬆️ Set n8n-workflow@0.92.0 on n8n-core 🔖 Release n8n-core@0.110.0 ⬆️ Set n8n-core@0.110.0 and n8n-workflow@0.92.0 on n8n-node-dev 🔖 Release n8n-node-dev@0.49.0 ⬆️ Set n8n-core@0.110.0 and n8n-workflow@0.92.0 on n8n-nodes-base 🔖 Release n8n-nodes-base@0.167.0 🔖 Release n8n-design-system@0.15.0 ⬆️ Set n8n-design-system@0.15.0 and n8n-workflow@0.92.0 on n8n-editor-ui 🔖 Release n8n-editor-ui@0.136.0 ⬆️ Set n8n-core@0.110.0, n8n-editor-ui@0.136.0, n8n-nodes-base@0.167.0 and n8n-workflow@0.92.0 on n8n 🔖 Release n8n@0.169.0 📚 Update CHANGELOG.md 📚 Fix CHANGELOG.md file ⚡ Add Odoo and RedisTrigger node codex (n8n-io#3005) * .168.2fixed: Auto stash before rebase of "refs/heads/codex/0.168.2fixed" Odoo and Redis Trigger codex files update * Update RedisTrigger.node.json :zap: Add KoBoToolbox and Linear codex files (n8n-io#3040) KoBoToolbox KoBoToolbox Trigger Linear :books: Add missing full stop to license text * (fix): Added missing full stop to license GitHub does not render the single line breaks in the *Limitations* section. The added full stop makes it easier to read our license. * :books: Add also to other files fix(AWS Lambda Node): Fix "Invocation Type" > "Continue Workflow" (n8n-io#3010) * 🔨 fix for running in continue workflow * ⚡ Minor simplification 📚 Add one more missing full stop to license text fix(core): Add logs and error catches for possible failures in queue mode (n8n-io#3032) fix(Supabase Node): Fix Row > Get operation (n8n-io#3045) fix(Supabase Node): Send token also via Authorization Bearer (n8n-io#2814) Send Authorization Bearer in headers Fix typo in validateCredentials function fix(Wise Node): Fix issue when executing a transfer (n8n-io#3039) :zap: Fix credentials import success message (n8n-io#3038) :books: Add missing full stop to license text (n8n-io#3028) Adding "." L15. In addition, the markdown display don't show line break as in the editor. :books: Add note to changelog linking to historic log (n8n-io#3031) feat(HTTP Request Node): Add support for OPTIONS method (n8n-io#3030) fix(Xero Node): Fix some operations and add support for setting address and phone number (n8n-io#3048) * 🐛 Fix issue when sending Organization ID - Xero node * 👕 Fix linting issue feat(Crypto Node): Add Generate operation to generate random values (n8n-io#2541) * ✨ Add generate action to crypto node * ⚡ small fixes, nodelinter issues fixes * ⚡ Improvements * ⚡ Fix order feat(Reddit Node): Add possibility to query saved posts (n8n-io#3034) * chore: add nvmrc with required node version * feat: added saved posts to reddit node with credentials on User resource * Changed Details order * Fixed lint issue * Moved saved posts to profile as it only works for the logged in user, This avoids the breaking change * Removed .nvmrc * ⚡ Improvements feat(Jira Node): Add Simplify Output option to Issue > Get (n8n-io#2408) * ✨ Add option to use Jira field display names * 🚸 Make mapped fields more deterministic * ♻️ Refactor Jira user loadOptions * Moved and renamed the option as well as only returning the fields to * Tweaked Friendly Fields to make it "Simplify Output" following similar patterns to other nodes * ⚡ Improvements feat(Zendesk Node): Add ticket status "On-hold" 🔖 Release n8n-workflow@0.93.0 ⬆️ Set n8n-workflow@0.93.0 on n8n-core 🔖 Release n8n-core@0.111.0 ⬆️ Set n8n-core@0.111.0 and n8n-workflow@0.93.0 on n8n-node-dev 🔖 Release n8n-node-dev@0.50.0 ⬆️ Set n8n-core@0.111.0 and n8n-workflow@0.93.0 on n8n-nodes-base 🔖 Release n8n-nodes-base@0.168.0 🔖 Release n8n-design-system@0.16.0 ⬆️ Set n8n-design-system@0.16.0 and n8n-workflow@0.93.0 on n8n-editor-ui 🔖 Release n8n-editor-ui@0.137.0 ⬆️ Set n8n-core@0.111.0, n8n-editor-ui@0.137.0, n8n-nodes-base@0.168.0 and n8n-workflow@0.93.0 on n8n 🔖 Release n8n@0.170.0 ⬆️ Update package-lock.json file 📚 Update CHANGELOG.md with version 0.170.0 Co-Authored-By: Jonathan Bennetts <jonathan.bennetts@gmail.com> Co-Authored-By: ricardo <ricardoespinoza105@gmail.com> Co-Authored-By: Manuel [tennox] <2084639+tennox@users.noreply.github.com> Co-Authored-By: Justin Halter <jhalter@weare5stones.com> Co-Authored-By: Yann Jouanique <yann.jouanique@oneacrefund.org> Co-Authored-By: Jan Oberhauser <jan.oberhauser@gmail.com> Co-Authored-By: Marcin Koziej <marcin@cahoots.pl> Co-Authored-By: Niv <nivbelleli@gmail.com> Co-Authored-By: michael-radency <michael.k@radency.com> Co-Authored-By: Yassine Fathi <hi@m4tt72.com>

Yann-J added 3 commits December 1, 2021 17:56

First version

a4db868

Added hooks

c91be30

Added Credentials test

63c805d

Yann-J added 5 commits December 2, 2021 12:22

Add support for downloading attachments

46823c6

Slight restructure of downloaded binaries

05ba5b6

Added Trigger node

57ec535

Some linting

6053291

Reverting package-lock changes

9ef4f2b

Yann-J changed the title ~~Kobotoolbox node~~ 📱Kobotoolbox API and Trigger nodes Dec 3, 2021

Minor GeoJSON parsing fixes

f658941

Yann-J mentioned this pull request Dec 5, 2021

KoboToolbox node docs n8n-io/n8n-docs#678

Closed

Yann-J added 7 commits December 5, 2021 22:09

KoboToolbox: improve GeoJSON format

f55fb62

Kobo: Support for get/set validation status

6ebc527

Remove some logs

146fee8

Merge branch 'master' into kobotoolbox

b32153d

[kobo] Fix default attachment options

a87c9d2

Proper debug logging

be9dbc6

Support for hook log status filter

2469c9c

ivov added node/new Creation of an entirely new node community Authored by a community member labels Dec 15, 2021

csuermann requested a review from RicardoE105 January 14, 2022 08:40

RicardoE105 reviewed Jan 14, 2022

View reviewed changes

Kobo: Review fixes

a623e90

[kobo] Update branded spelling

17a8e12

RicardoE105 requested changes Feb 2, 2022

View reviewed changes

[kobo] Support pagination

575f733

RicardoE105 added a commit that referenced this pull request Feb 7, 2022

⚡ Improvements to #2510

e17b82f

[kobo] Fix linting error

9d550f2

Yann-J added 2 commits February 8, 2022 13:09

[kobo] Load hook IDs

ee1ea4b

[kobo] Lookup hooks

edac21a

nivbe06 closed this Mar 31, 2022

📱Kobotoolbox API and Trigger nodes #2510

📱Kobotoolbox API and Trigger nodes #2510

Conversation

Yann-J commented Dec 1, 2021

CLAassistant commented Dec 1, 2021 • edited Loading

Yann-J commented Dec 3, 2021

Yann-J commented Dec 14, 2021

Yann-J commented Dec 15, 2021

Yann-J commented Jan 12, 2022

csuermann commented Jan 14, 2022

RicardoE105 left a comment

Choose a reason for hiding this comment

Yann-J commented Jan 18, 2022

Yann-J commented Jan 21, 2022

RicardoE105 commented Jan 24, 2022

RicardoE105 commented Jan 24, 2022

RicardoE105 commented Jan 24, 2022

RicardoE105 commented Jan 24, 2022

Yann-J commented Jan 25, 2022

RicardoE105 commented Feb 2, 2022

Yann-J commented Feb 2, 2022

RicardoE105 commented Feb 2, 2022 • edited Loading

RicardoE105 Jan 24, 2022

Choose a reason for hiding this comment

Yann-J Feb 3, 2022

Choose a reason for hiding this comment

RicardoE105 Jan 24, 2022

Choose a reason for hiding this comment

RicardoE105 Jan 24, 2022

Choose a reason for hiding this comment

RicardoE105 Jan 24, 2022

Choose a reason for hiding this comment

RicardoE105 Jan 24, 2022

Choose a reason for hiding this comment

Yann-J Feb 3, 2022

Choose a reason for hiding this comment

Yann-J commented Feb 3, 2022

Yann-J commented Feb 3, 2022

Yann-J commented Feb 3, 2022

RicardoE105 commented Feb 7, 2022

Yann-J commented Feb 8, 2022

Yann-J commented Feb 9, 2022

RicardoE105 commented Feb 15, 2022

Yann-J commented Feb 16, 2022

Yann-J commented Mar 3, 2022

RicardoE105 commented Mar 7, 2022

janober commented Mar 20, 2022

CLAassistant commented Dec 1, 2021 •

edited

Loading

RicardoE105 commented Feb 2, 2022 •

edited

Loading