Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-269] Add some unit tests for PostgreSQL #1616

Closed
wants to merge 1 commit into from

Conversation

sekikn
Copy link
Contributor

@sekikn sekikn commented Jun 22, 2016

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

Testing Done:

  • ran ./run_unit_tests.sh tests.operators:PostgresTest and confirmed all 4 tests passed

There is only one unit test for PostgreSQL for now.
This patch add some tests as well as MySQL.

@criccomini
Copy link
Contributor

Looks like tests are failing

root: INFO: CREATE TABLE IF NOT EXISTS test_postgres_to_postgres (LIKE INFORMATION_SCHEMA.TABLES)
root: ERROR: inherited relation "tables" is not a table
Traceback (most recent call last):
  File "/home/travis/build/apache/incubator-airflow/airflow/models.py", line 1324, in run
    result = task_copy.execute(context=context)
  File "/home/travis/build/apache/incubator-airflow/airflow/operators/generic_transfer.py", line 75, in execute
    destination_hook.run(self.preoperator)
  File "/home/travis/build/apache/incubator-airflow/airflow/hooks/dbapi_hook.py", line 145, in run
    cur.execute(s)
ProgrammingError: inherited relation "tables" is not a table

@sekikn
Copy link
Contributor Author

sekikn commented Jun 28, 2016

I've missed that error since it doesn't occur in my local environment. Thanks for the comment! I'm not sure its cause for now, but I'll investigate it.

@sekikn
Copy link
Contributor Author

sekikn commented Jun 28, 2016

After some experiments, I've figured out the cause. information_schema.tables is a view in PostgreSQL, and PostgreSQL 9.1 probably doesn't support copying tables by "CREATE TABLE LIKE" from views.
The following sentence appears in 9.2 document for the first time, and not in 9.1.

The LIKE clause can also be used to copy columns from views, foreign tables, or composite types.

Possible solutions are:

  1. specifying PostgreSQL version greater than 9.1 in .travis.yml
  2. instead of a view, using a real table (such as pg_class, information_schema.sql_languages or something) as a copying source

@criccomini What version of PostgreSQL does Airflow officially support? If 9.1 is not supported, I prefer the first option for consistency with MySQL's test case.

@bolkedebruin
Copy link
Contributor

9.1 is relative old, so I am ok with a more recent version.

There is only one unit test for PostgreSQL for now.
This patch add some tests as well as MySQL.
@sekikn
Copy link
Contributor Author

sekikn commented Jun 30, 2016

Thanks! I've just set postgresql version to 9.2 on Travis CI and made the PR squashed.

@sekikn
Copy link
Contributor Author

sekikn commented Jun 30, 2016

The latest CI failure is irrelevant to this issue. The unit test in question succeeded as follows:

INFO  [root] DROP TABLE IF EXISTS test_postgres_to_postgres
INFO  [root] CREATE TABLE IF NOT EXISTS test_postgres_to_postgres (LIKE INFORMATION_SCHEMA.TABLES)
INFO  [root] Inserting rows into postgres_default
INFO  [root] Using connection to: localhost
INFO  [root] Done loading. Loaded a total of 100 rows
ok

@criccomini
Copy link
Contributor

LGTM

Thanks so much for adding tests. Really valuable.

@asfgit asfgit closed this in a59ba04 Jun 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants