New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fixes for views: return views in get_table_options and improve reflection #46

Merged
merged 1 commit into from Sep 16, 2015

Conversation

Projects
None yet
2 participants
@jklukas
Collaborator

jklukas commented Sep 9, 2015

Because SQLAlchemy only has limited support for views, some methods that include table in the name are also supposed to return views.

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 9, 2015

Collaborator

Since there's little distinction between tables and views, this PR also includes a refactor to reduce duplication of code between table-specific and view-specific methods.

Collaborator

jklukas commented Sep 9, 2015

Since there's little distinction between tables and views, this PR also includes a refactor to reduce duplication of code between table-specific and view-specific methods.

@jklukas jklukas changed the title from Bug fix: return views in get_table_options to Bug fixes for views: return views in get_table_options and improve reflection Sep 10, 2015

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 11, 2015

Collaborator
Collaborator

jklukas commented Sep 11, 2015

@@ -643,7 +623,8 @@ def _get_all_table_and_view_info(self, connection, **kw):
AS "diststyle",
c.relowner AS "owner_id",
u.usename AS "owner_name",
pg_get_viewdef(c.oid) AS "view_definition",
TRIM(TRAILING ';' FROM pg_catalog.pg_get_viewdef(c.oid, true))

This comment has been minimized.

@graingert

graingert Sep 11, 2015

Collaborator

What's this trim for?

@graingert

graingert Sep 11, 2015

Collaborator

What's this trim for?

This comment has been minimized.

@jklukas

jklukas Sep 11, 2015

Collaborator

The output of pg_get_viewdef includes a semicolon at the end of the query. The SQLAlchemy CreateTable, etc. commands don't include a semicolon at the end of the output, so we need to strip that out somewhere for consistency.

This TRIM(TRAILING ';' is the solution used in psql's \d+ command to get rid of the semicolon.

@jklukas

jklukas Sep 11, 2015

Collaborator

The output of pg_get_viewdef includes a semicolon at the end of the query. The SQLAlchemy CreateTable, etc. commands don't include a semicolon at the end of the output, so we need to strip that out somewhere for consistency.

This TRIM(TRAILING ';' is the solution used in psql's \d+ command to get rid of the semicolon.

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 11, 2015

Collaborator

This doesn't change functionality at all right?

Collaborator

graingert commented Sep 11, 2015

This doesn't change functionality at all right?

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 11, 2015

Collaborator

This doesn't change functionality at all right?

With this change, get_view_definition now doesn't include a semicolon in its output and get_table_options now returns an empty dict when the input relation is a view (previously, this raised an exception).

Both of the previous behaviors I consider bugs.

Collaborator

jklukas commented Sep 11, 2015

This doesn't change functionality at all right?

With this change, get_view_definition now doesn't include a semicolon in its output and get_table_options now returns an empty dict when the input relation is a view (previously, this raised an exception).

Both of the previous behaviors I consider bugs.

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 11, 2015

Collaborator

Can you add tests and changelog for these bugs please?

Collaborator

graingert commented Sep 11, 2015

Can you add tests and changelog for these bugs please?

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 11, 2015

Collaborator

I'm going to need some help if we're going to try to add tests for these. The current test structure for reflection relies on declarative_base objects, which AFAIK can't create views.

@graingert - Do you have suggestions about how we can create a view in our test models to reflect from?

Collaborator

jklukas commented Sep 11, 2015

I'm going to need some help if we're going to try to add tests for these. The current test structure for reflection relies on declarative_base objects, which AFAIK can't create views.

@graingert - Do you have suggestions about how we can create a view in our test models to reflect from?

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 11, 2015

Collaborator

One option is to use the redshift_engine pytest fixture and create views
on that using raw SQL. Beware this fixture is slower than
redshift_session because it generates a new database for each test
function.
On 11 Sep 2015 21:51, "Jeff Klukas" notifications@github.com wrote:

I'm going to need some help if we're going to try to add tests for these.
The current test structure for reflection relies on declarative_base
objects, which AFAIK can't create views.

@graingert https://github.com/graingert - Do you have suggestions about
how we can create a view in our test models to reflect from?


Reply to this email directly or view it on GitHub
#46 (comment)
.

Collaborator

graingert commented Sep 11, 2015

One option is to use the redshift_engine pytest fixture and create views
on that using raw SQL. Beware this fixture is slower than
redshift_session because it generates a new database for each test
function.
On 11 Sep 2015 21:51, "Jeff Klukas" notifications@github.com wrote:

I'm going to need some help if we're going to try to add tests for these.
The current test structure for reflection relies on declarative_base
objects, which AFAIK can't create views.

@graingert https://github.com/graingert - Do you have suggestions about
how we can create a view in our test models to reflect from?


Reply to this email directly or view it on GitHub
#46 (comment)
.

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 14, 2015

Collaborator

Test failure is an unexpected 500 from bigcrunch again.

Collaborator

jklukas commented Sep 14, 2015

Test failure is an unexpected 500 from bigcrunch again.

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 14, 2015

Collaborator

For now you'll have to retry the build after redshift has booted up properly
On 14 Sep 2015 16:44, "Jeff Klukas" notifications@github.com wrote:

Test failure is an unexpected 500 from bigcrunch again.


Reply to this email directly or view it on GitHub
#46 (comment)
.

Collaborator

graingert commented Sep 14, 2015

For now you'll have to retry the build after redshift has booted up properly
On 14 Sep 2015 16:44, "Jeff Klukas" notifications@github.com wrote:

Test failure is an unexpected 500 from bigcrunch again.


Reply to this email directly or view it on GitHub
#46 (comment)
.

.compile(engine))
def test_view_reflection(redshift_engine):

This comment has been minimized.

@graingert

graingert Sep 15, 2015

Collaborator

nice

@graingert

graingert Sep 15, 2015

Collaborator

nice

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 15, 2015

Collaborator

@bouk @cpcloud @jklukas I'm pro merging this, once it has a changelog entry

Collaborator

graingert commented Sep 15, 2015

@bouk @cpcloud @jklukas I'm pro merging this, once it has a changelog entry

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 15, 2015

Collaborator

Totally forgot about the changelog. I just pushed an update that adds a note about the changes.

Collaborator

jklukas commented Sep 15, 2015

Totally forgot about the changelog. I just pushed an update that adds a note about the changes.

@graingert

View changes

Show outdated Hide outdated CHANGES.rst
@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 16, 2015

Collaborator

We're still getting intermittent 500 responses from bigcrunch. How well do you understand the behavior, @graingert?

Collaborator

jklukas commented Sep 16, 2015

We're still getting intermittent 500 responses from bigcrunch. How well do you understand the behavior, @graingert?

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 16, 2015

Collaborator

@jklukas there's a small window between the cluster booting and cluster being available (DNS etc) where these 500s occur. I know how to fix this. Are you able to access the rebuild button on Travis?

Collaborator

graingert commented Sep 16, 2015

@jklukas there's a small window between the cluster booting and cluster being available (DNS etc) where these 500s occur. I know how to fix this. Are you able to access the rebuild button on Travis?

@jklukas

This comment has been minimized.

Show comment
Hide comment
@jklukas

jklukas Sep 16, 2015

Collaborator

Are you able to access the rebuild button on Travis?

I have limited experience with Travis, so didn't know to look for one. I was just able to trigger a rebuild. We'll see how it does.

Collaborator

jklukas commented Sep 16, 2015

Are you able to access the rebuild button on Travis?

I have limited experience with Travis, so didn't know to look for one. I was just able to trigger a rebuild. We'll see how it does.

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Sep 16, 2015

Collaborator

This looks great, merging. Thanks very much!

Collaborator

graingert commented Sep 16, 2015

This looks great, merging. Thanks very much!

graingert added a commit that referenced this pull request Sep 16, 2015

Merge pull request #46 from graingert/view-fixes
Bug fixes for views: return views in get_table_options and improve reflection

@graingert graingert merged commit c746d20 into master Sep 16, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@graingert graingert deleted the view-fixes branch Sep 16, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment