New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reflection regression #64
Conversation
@berdario - Am I correct in understanding that the problem you've seen is that databases in alternate schemas don't get reflected properly? I have successfully used the most recent version of the dialect to reflect tables from schemas other than public, so I'm trying to figure out exactly what the failure mode is. I'm not sure whether the failing test here is due to a defect of the dialect, or a defect of the test framework. |
Yup, the problem I've seen is that tables don't get reflected properly... The outermost error I get is Since I found the problem here with a schema/table that is actually in use I think it's perfectly normal that the test fails, and thus it should be a defect in the dialect |
@berdario: In your case, is the schema you're reflecting a table from not in the |
Indeed, if I execute |
I think this is the important gotcha (from http://docs.aws.amazon.com/redshift/latest/dg/r_PG_TABLE_DEF.html):
I'm looking at some options for solving this. |
05fc0f5
to
c22f012
Compare
-- We temporarily set the search_path here to include | ||
-- all non-system schemas because pg_table_def doesn't show | ||
-- results for schemas not on the search_path. | ||
SET LOCAL search_path TO {search_path}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's ridiculous that we have to do this, but it works and I think it'll be robust.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So what was postgres default reflection doing before to get the tables?
I just pushed an amended commit that actually fixes #62. @berdario's test case now passes; again, thank you for reporting the issue and writing the test. @berdario and @graingert - Does this look good to you? |
Does this table https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_TABLE_INFO.html have anything useful in? |
Otherwis have a look at https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminScripts/table_info.sql ? |
My understanding of the situation is the
The PG dialect doesn't have this visibility problem because it doesn't have to access the You can see that AWS also needs to manipulate the search_path in their column encoding utility, which I take as confirmation that search_path manipulation is the only approach here that's going to be comprehensive. The |
Is this connection always a connection? Could it be an engine? I think you might need to use a nested transaction here |
c22f012
to
94723ec
Compare
Added a nested transaction per @graingert's recommendation. |
94723ec
to
982283e
Compare
Removed |
There's a lot of discussion happening on 94723ec which isn't automatically pulled in here anymore because the commit has been amended. |
ah nuts I meant to comment on the PR |
982283e
to
f7e6ecd
Compare
As discussed in 94723ec, we now use a There's also now a test to verify that the search_path change doesn't leak. |
Note that since we now set the search_path back, there's no need for ensuring a transaction, so I'm not explicitly creating one. |
That makes sense, I'm happy for you to merge this when you're ready |
This When I remove it ( |
What also works is putting the
|
Iteration on #62. Because this branch is on the main repo, travis tests should run.
This adds creation of the schema, but the new test fails with
NoSuchTableError: other_schema.basic
. I don't understand why this happens.cc @berdario, @graingert