Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document schema size limitation in Postgres Source connector docs #14230

Closed
grishick opened this issue Jun 28, 2022 · 4 comments
Closed

Document schema size limitation in Postgres Source connector docs #14230

grishick opened this issue Jun 28, 2022 · 4 comments
Assignees
Labels
area/documentation Improvements or additions to documentation autoteam team/documentation type/enhancement New feature or request

Comments

@grishick
Copy link
Contributor

Tell us about the documentation you'd like us to add or update.

Platform is currently not able to handle schemas larger than 4MB. This is a platform-wide limitation. However, it should be documented for GA connectors that are likely to produce such large schemas.

If applicable, add links to the relevant docs that should be updated

@grishick grishick added area/documentation Improvements or additions to documentation type/enhancement New feature or request labels Jun 28, 2022
@grishick grishick self-assigned this Jun 28, 2022
@grishick
Copy link
Contributor Author

To close this issue:

  • identify the exact size limit
  • create an approximate estimate in the form of "number of tables with average number of columns per table" that reflects the upper limit that source postgres connector can work with

@kimerinn
Copy link
Contributor

kimerinn commented Jul 8, 2022

@grishick , should this task be closed? I could not find any info about postgres table metadata size, so unfortunately, I could not define max number of tables/columns per table for 4Mb limit

@grishick
Copy link
Contributor Author

grishick commented Jul 8, 2022

@kimerinn could you do some testing to find an approximate number of columns x tables that would hit the limit? I could expect that if a schema is approaching some number X of columns across all tables it will start hitting this limit.

@kimerinn
Copy link
Contributor

kimerinn commented Jul 11, 2022

@grishick I added some rough calculations that I found playing with my Windows PostgreSQL 12. Review please
#14602

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Improvements or additions to documentation autoteam team/documentation type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants