Accelerated need to limit large url requests to “list records”, with alternative POST version

[FredZhao-at]

Hi! Fred here again, from the Airtable API team. Another update is that as part of recent infrastructure improvements to airtable.com, we needed to accelerate enforcement of rejecting any requests whose request URLs exceed 16k characters. The changes described below are in effect today.

• Q: What endpoints does this affect?
  • The 16k character limit applies to all current and future API request URLs
  • Based on our internal audit of current usage, this primarily impacts the “list records” endpoint, which is a GET request to /v0/{baseId}/{tableIdOrName} (https://airtable.com/developers/web/api/list-records)
• Q: What is the impact of this change?
  • The vast majority of our API requests have remained well under the 16k character limit, and would not be affected
  • However, requests exceeding this limit will now be rejected, so we expect some calls to now require an update to handle the large cases. For example, requests whose query parameter for filterByFormula is long enough to make the overall URL length to exceed that character limit
  • Therefore, we recommend either switching entirely to the alternative detailed below, or switching to it when you can detect that a request URL will exceed the 16k character limit
• Q: What is the alternative?
  • Use the latest version of https://github.com/Airtable/airtable.js
  • For gatsby-source-airtable: Consider bumping the airtable dependency to the latest version. If you aren't using the filterByFormula parameter, this change might not impact you. But there could be an edge case where a base name or view name causes this limit to be exceeded
• Q: Why the urgency for this change?
  • I apologize for the inconvenience due to the accelerated timeline here. While we originally planned to start this enforcement in January 2023, we needed to deploy this change recently this week, as part of required infrastructure improvements to maintain the stability of our public API

[ZeldOcarina]

@FredZhao-at same here, no answer, this may be why I am not getting some records anymore..

[jbolda]

@FredZhao-at I was on extended vacation, not actively involved with Gatsby anymore, and do not receive any compensation for this so I have personally attempted to maintain it at the bare minimum. Unfortunately, it sounds like this was pretty short notice.

Is there interest in Airtable proper taking a direct ownership stake in maintaining this plugin?

[ZeldOcarina]

Hi @jbolda thank you for answering here and for the incredible efforts you have put in on the development of this plugin!

This has now become a super core part of my company activities, we have dozens of websites using this on heavy duty so I am really interested in it being mantained and I proposed to help on this.

Please take a look at this discussion, I am not sure if it would be a viable option for you! :)

Sorry if this might be OT in this issue.

[FredZhao-at]

@jbolda thanks for taking a look, and happy new year!

From the Airtable side, our goal is to ensure that https://github.com/Airtable/airtable.js is up-to-date so that other packages can rely on it, so unfortunately we won't be able to take on ownership of specific plugins that in turn use airtable.js.

That being said, the amount of work needed here should be minimal:

1. First of all, if this package isn't using the filterByFormula parameter, this change might not impact it at all. We're leaving that decision to each package maintainer to make the call
2. If this change could impact you, the fix is also very easy: just bump up the package dependency version. Again, we're also leaving that to each package maintainer

[jbolda]

I have elected to move this plugin into the GUC, and these issues may be best addressed there.