Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: ignore tables for PgDatabaseMetaData.getTypeInfo #1302

Merged
merged 1 commit into from Oct 16, 2018

Conversation

@davecramer
Copy link
Member

@davecramer davecramer commented Sep 19, 2018

postgres considers tables to be composite types which
in very large schemas with lots of tables slows down
getTypeInfo.
This fix filters out tables from types

fixes #1301

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Does mvn checkstyle:check pass ?

Changes to Existing Features:

  • Does this break existing behaviour? If so please explain.
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully run tests with your changes locally?
postgres considers tables to be composite types which
in very large schemas with lots of tables slows down
getTypeInfo.
This fix filters out tables from types
@bokken
Copy link
Member

@bokken bokken commented Oct 16, 2018

+1

@davecramer davecramer merged commit e44e4e8 into pgjdbc:master Oct 16, 2018
1 check passed
@davecramer davecramer deleted the typecache branch Oct 16, 2018
@camelcamro
Copy link

@camelcamro camelcamro commented Mar 20, 2019

we are still having this issue:
should that be fixed on postgres driver version (42.2.5) ?
here the query which was hanging and is blocking the whole database to run the vacuum, and LIVE_TUp are growing up endless ttil the crash.
any fix for it ?
or reopen the ticket ?
cu camel

hanging query:
"SELECT typinput='array_in'::regproc, typtype FROM pg_catalog.pg_type LEFT JOIN (select ns.oid as nspoid, ns.nspname, r.r from pg_namespace as ns join ( select s.r, (current_schemas(false))[s.r] as nspname from generate_series(1, array_upper(current_schemas(false), 1)) as s(r) ) as r using ( nspname ) ) as sp ON sp.nspoid = typnamespace WHERE typname = $1 ORDER BY sp.r, pg_type.oid DESC LIMIT 1"

@davecramer
Copy link
Member Author

@davecramer davecramer commented Mar 20, 2019

@camelcamro This does not look like the same query causing the original problem in this ticket. Please open a new ticket

Looking at this

sql = "SELECT typinput='array_in'::regproc, typtype "

, it should actually return pretty quickly. How many schemas do you have ?

@thejames97
Copy link

@thejames97 thejames97 commented Jul 18, 2019

Even with this fix, I am getting thousands of results and it causes a significant slowdown when Hibernate is pulling types during initialization. I found this query that is even more targeted (based on this SO post: https://stackoverflow.com/a/16349665/10157442):

SELECT t.typname,t.oid
FROM pg_catalog.pg_type t
JOIN pg_catalog.pg_namespace n ON (t.typnamespace = n.oid)
WHERE n.nspname != 'pg_toast'
AND (t.typrelid = 0 OR (SELECT c.relkind = 'c' FROM pg_catalog.pg_class c WHERE c.oid = t.typrelid))
AND NOT EXISTS(SELECT 1 FROM pg_catalog.pg_type el WHERE el.oid = t.typelem AND el.typarray = t.oid)
AND pg_catalog.pg_type_is_visible(t.oid);

With this query, I still get back 201 types, but that's much more reasonable than the 10s of thousands the current query returns for my DB.

@davecramer
Copy link
Member Author

@davecramer davecramer commented Jul 19, 2019

@thejames97 The spec for getTypeInfo says get all of the types. Why does your database have 10's of thousands of types ?

@thejames97
Copy link

@thejames97 thejames97 commented Jul 19, 2019

@davecramer Ha! That's what I was wondering! I'm new to Postgres and I'm not the DBA. But I can't imagine there are really more than about the 28 or so actual types that Oracle's getTypeInfo returns. Maybe as many as 200, but even that seems a stretch.

I have things like this, which make perfect sense:
bool, bytea, char, int8, text, oid, json, xml

But then I also get a bunch of stuff like this:
_bool, _bytea, _char (everything repeats with underscore prefix)

But most of what I am seeing looks like views and functions and possibly other object types. I also see them repeated with underscore prefixes...I even see tables with underscore prefixes which did not get filtered out.

According to the notes on this fix, the updated query was supposed to filter out tables from the getTypeInfo query, but are other object types still being included? Is everything being repeated with an underscore prefix?

BTW, this is a very large DB, so if all the user defined objects (tables, views, functions, sequences, etc) are being returned as types (twice!), then that explains why I am seeing such a huge result set coming back and a fairly significant delay in my microservices' startup times.

@davecramer
Copy link
Member Author

@davecramer davecramer commented Jul 19, 2019

the fix should remove tables, views, functions, sequences. The objects with _ are normal as those are arrays of the types

davecramer added a commit to davecramer/pgjdbc that referenced this issue Jul 5, 2021
PostgreSQL considers tables to be composite types which
in very large schemas with lots of tables slows down getTypeInfo.
This fix filters out tables from types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

5 participants