BREAKING CHANGE(core): Introduce response.data with support for form-urlencoded and custom parsing

[FokkeZB]

Definitions:

• Code-mode: FunctionSchema
• Form-mode: RequestSchema

Commits

I've split the work up in 3 commits:

• Changes: 44f509a
• Tests: 5bb92a2
• Documentation & Examples: 439a596

Changes

• prepareContentResponse now parses .content to .data if its either application/x-www-form-urlencoded or can be parsed as JSON.
• prepareContentResponse still sets .json but it should be considered deprecated.
• executeHttpRequest now uses .data instead of (again) parsing .content as application/x-www-form-urlencoded or JSON.
• executeHttpRequest now throws a helpful error instead of SyntaxError if .data is not set and thus not application/x-www-form-urlencoded, JSON or set in afterResponse.

See the first commit.

Benefits

• Code-mode no longer need to parse application/x-www-form-urlencoded, just like form-mode requests.
• Developers don't need to JSON.stringify() their custom-parsed .content back to the same property for form-mode to work.
• The type-agnostic .data property allows us to later add support for types like XML.

See the below Examples.

Examples

These examples all use afterResponse. The code-mode ones could also handle this within the function of course, but this makes it easier to compare between code and form-mode.

Form-mode request with form response

Already and still works out of the box. It just now relies on response.data set by prepareResponse instead of parsing response.content just before returning it.

This means it may be a breaking change for CLI apps that were doing custom parsing in afterResponse as the Form-mode request with XML response example.

Code-mode request with form response

Before

const querystring = require('querystring');

const afterResponse = (response) => {
  if (response.headers.get('content-type') === 'application/x-www-form-urlencoded') {
    response.json = querystring.parse(response.content);
  }
}

After

z.request('https://example.com/').then(response => response.data);

Code-mode request with JSON response

You can still use response.json but we encourage switching to response.data.

Before

z.request('https://example.com/').then(response => response.json);

After

z.request('https://example.com/').then(response => response.data);

Form-mode request with XML response

Note that both before and after currently are only possible in CLI apps, but once https://github.com/zapier/zapier/pull/39899 makes the Middleware tab public for all, this can also be used in visual builder apps.

Before

As we defaulted to parse response.content as JSON:

const parser = require('fast-xml-parser');

const afterResponse = (response) => {
  if ('xml' in response.headers.get('content-type')) {
    const parsed = parser(response.content);
    response.content = JSON.stringify(parsed);
  }
}

After

As we now use response.data:

const parser = require('fast-xml-parser');

const afterResponse = (response) => {
  if ('xml' in response.headers.get('content-type')) {
    response.data = parser(response.content);
  }
}

Code-mode request with XML response

Before

const parser = require('fast-xml-parser');

const afterResponse = (response) => {
  if ('xml' in response.headers.get('content-type')) {
    response.json = parser(response.content);
  }
}

After:

const parser = require('fast-xml-parser');

const afterResponse = (response) => {
  if ('xml' in response.headers.get('content-type')) {
    response.data = parser(response.content);
  }
}

[FokkeZB]

@szchenghuang a heads-up that I'd like to include this in v10 as well. It's the last piece of the puzzle that makes afterResponse work well with shorthand/form-mode request. I'll have it ready for review this week.

[xavdid]

@FokkeZB we're getting v10 ready to go out the door - do you have a timeline on this?

[FokkeZB]

@xavdid I've got it almost finished. I was prioritizing https://github.com/zapier/zapier/pull/39898 over this one, but I now realize that we won't need to switch visual builder apps immediately, so this should come first as we don't want to have another breaking change soon after.

I'll work on this this afternoon and also prioritize writing docs on both with lots of examples.

[eliangcs]

@FokkeZB quick question: looks like response.json will continue to work, so is it still a breaking change?

[FokkeZB]

@eliangcs it would be breaking for CLI apps that currently change response.content in their afterResponse middleware, with the expectation for form-mode requests to pick up on that. They'll have to replace response.content = JSON.stringify(parsedOrTransformed) with response.data = parsedOrTransformed.

[FokkeZB]

self-review of changes

[FokkeZB]

This allows afterResponse to do any custom parsing and override response.data.

[FokkeZB]

self-review added tests

[FokkeZB]

This adds test to ensure that form-mode/shorthand request still work with both JSON and form-urlencoded. We didn't have any tests for it AFAIK :(

[FokkeZB]

This adds tests for prepareResponse for which AFAIK we didn't have any.

[FokkeZB]

self-review docs & examples

[FokkeZB]

Trello is a great example of an app that can now use response.data

[xavdid]

looks awesome! Thanks for putting this all together.

[eliangcs]

Looks good except for the changes in legacy-scripting-runner. We'll want legacy-scripting-runner to be backward compatible with older versions of core.

[FokkeZB]

Thanks for the reviews @xavdid and @eliangcs - all feedback has been addressed.