AirbyteLib: Fix column count mismatch bug

[aaronsteers]

This PR resolves a bug when a fully-null or fully-missing column causes the insert to fail, because the number of named columns in the destination table does not match the number of columns in the parquet files.

The actual implementation required a large bit of refactoring.

1. File writers now also get initialized with a catalog manager. This allows the file writer to make decisions based upon the catalog schema. In our case, this means detecting which properties are expected for the given stream so they can be appended.
2. Source's get_records() implementation needed similar treatment for adding any missed properties. Now both Lazy and Cached datasets will have all top-level keys present, even if the source omits them entirely.
3. I moved some methods like _get_stream_config() and _get_stream_json_schema() down from the SQLCache class into the underlying RecordProcessors base class, so that FileWriters can share the same code.

Drive-by changes:

1. source_catalog was deprecated from processors but not removed. Now it is removed.
2. Catalog managers' initialization is cleaned up. Only streams that are 'incoming' will actually have their catalog metadata written.
3. "streams with data" has been tightened up somewhat.
4. We had typed "pa.Table | pa.RecordBatch" in several places, even though the type is always pa.Table. Since only Table as the append_column() method, I've now made typing consistent across the board so there's no confusion about the expected type.
5. I've broken out the large register_source() method in the catalog manager into smaller methods: _update_catalog() and _save_catalog_to_internal_table().

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Feb 3, 2024 0:50am