You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like Airbyte to be able to ingest this data, currently we try to use the HTTP connector and we get:
E pydantic.error_wrappers.ValidationError: 1 validation error for AirbyteRecordMessage
E data
E value is not a valid dict (type=type_error.dict)
I see 3 potential solutions (thanks marcosmarxm for the idea to build something census specific):
Create a US Census specific connector, which supports their non-conventional responses
Patch the HTTP connector: Airbyte recognizes "list of list" as a special response format, assumes the first row is the column names, and returns something like:
@dmateusp happy to support you with creating a custom connector! please visit our Connector Development Kit to take a look at how you can get started. I also left a review on your PR.
Tell us about the problem you're trying to solve
I am trying to ingest public census data as part of a POC: https://api.census.gov/data/timeseries/eits/mrts/examples.html
But their responses are non standard: https://www.census.gov/data/developers/guidance/api-user-guide.Core_Concepts.html
e.g.
Describe the solution you’d like
I would like Airbyte to be able to ingest this data, currently we try to use the HTTP connector and we get:
I see 3 potential solutions (thanks marcosmarxm for the idea to build something census specific):
Are you willing to submit a PR?
Yes, I opened a PR that patches the HTTP connector, but I would like to attempt creating a census specific connector instead.
┆Issue is synchronized with this Asana task by Unito
The text was updated successfully, but these errors were encountered: