New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jdbc Operator How To #11472

Merged

mik-laj merged 12 commits into apache:master from francescomucio:master

Oct 29, 2020

Contributor

francescomucio commented Oct 12, 2020

Added an How To page for the Jdbc operator (and an example dag to get code snippet for the page)

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

francescomucio added 5 commits

October 8, 2020 01:08


          restore BREEZE.rst and created initialization function

cb89d9a


          moved intialize_airflow_variables inside initialize_common_environment

11dde77


          reverted back tenacity version

ea48276


          remove basestring

37d0786


          initial documentation for the JDBC Operator

b5eabc4

boring-cyborg bot added the area:docs label

github-actions bot commented Oct 12, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

github-actions bot commented Oct 12, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

github-actions bot commented Oct 12, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

francescomucio added 3 commits

October 12, 2020 21:53


          fixed typo

15a453c


          added jdbc label for the connection

33d5d70


          fixed typo

57f7e0f

github-actions bot commented Oct 13, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

github-actions bot commented Oct 13, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

github-actions bot commented Oct 13, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

francescomucio added 2 commits

October 13, 2020 10:28


          added label for jdbc how to in the operators-and-hooks-red.rst

93005e2


          trying to fix the link to jdbc connection

a294bcc

mik-laj reviewed

View reviewed changes

docs/howto/operator/jdbc.rst Outdated

+                * Install the python module jaydebeapi:
+                  .. code-block:: bash
+                    pip install jaydebeapi

Member

mik-laj Oct 13, 2020

I think, we should use extra packages to manage dependencies for Airflow. WDYT?
See: https://airflow.readthedocs.io/en/latest/installation.html#extra-packages

Contributor Author

francescomucio Oct 14, 2020

In the software section like for ODBC?
Should I add this as part of this PR?

mik-laj reviewed

View reviewed changes

docs/howto/operator/jdbc.rst Outdated

+              The parameter ``autocommit`` if set to ``True`` will execute a commit after
+              each command (default is ``False``)
+              Prerequisite Tasks

Member

mik-laj Oct 13, 2020

This should be the second section in the document and the first should describe the JDBC.

Contributor Author

francescomucio Oct 21, 2020

moved

mik-laj reviewed

View reviewed changes

docs/howto/operator/jdbc.rst Outdated


		pip install jaydebeapi

		* Install a JVM.

Member

mik-laj Oct 13, 2020

Can you add a link to external docs about it?

Contributor Author

francescomucio Oct 14, 2020

I always found installing Java very platform specific, what do you think of https://adoptopenjdk.net/installation.html?
I guess it is missing the step about setting the JAVA_HOME folder like you do in your Dockerfile.

mik-laj reviewed

View reviewed changes

docs/howto/operator/jdbc.rst Outdated

+                    pip install jaydebeapi
+                * Install a JVM.
+                * Have the JDBC driver for your database installed.

Member

mik-laj Oct 13, 2020

Can you add a link to external docs about it? Ideally, you should include a simple command here to check if this condition is met.

Contributor Author

francescomucio Oct 21, 2020

I have included a simple check, not sure if it is enough


          added more details about installing JVM and a test for the prerequisi…

edd3e5e

…tes setup

kaxil reviewed

View reviewed changes

airflow/example_dags/example_jdbc_operator.py Outdated

		@@ -0,0 +1,66 @@
		#

Member

kaxil Oct 22, 2020

This file should be in airflow.providers.jdbc.example_dags

Contributor Author

francescomucio Oct 26, 2020

moved

mik-laj reviewed

View reviewed changes

airflow/example_dags/example_jdbc_operator.py Outdated

+                  # [START howto_operator_jdbc_template]
+                  delete_data = JdbcOperator(
+                      task_id='insert_data',

Member

mik-laj Oct 22, 2020

Suggested change

      
                    task_id='insert_data',
          
                    task_id='delete_data',

Contributor Author

francescomucio Oct 26, 2020

thanks, fixed

mik-laj reviewed

View reviewed changes

docs/howto/operator/jdbc.rst

+                * Install the python module jaydebeapi:
+                  .. code-block:: bash
+                    pip install jaydebeapi

Member

mik-laj Oct 22, 2020

Suggested change

      
                  pip install jaydebeapi
          
                  pip install 'apache-airflow[jdbc]'

Contributor Author

francescomucio Oct 26, 2020

modified

mik-laj reviewed

View reviewed changes

docs/operators-and-hooks-ref.rst Outdated

@@ @@ -75,6 +75,8 @@ Fundamentals @@
                  * - :mod:`airflow.operators.latest_only`
                    -
+                 * - :mod:`airflow.providers.jdbc.operators.jdbc`

Member

mik-laj Oct 22, 2020

The fundamentals section only includes operators that are in the Airflow core. Please look at the protocol section: https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#protocol-operators-and-hooks

Contributor Author

francescomucio Oct 26, 2020

Thanks for pointing out this, I removed this and moved the link to the JDBC doc below


          moved example dags to the right location, fixed docs

102d295

mik-laj approved these changes

View reviewed changes

mik-laj merged commit ba9c044 into apache:master

Member

kaxil commented Oct 29, 2020

Good work @francescomucio 🎉

Member

ashb commented Oct 29, 2020

This, I think, has broken master and all open PRs

>       self.assertEqual([], invalid_files)
E       AssertionError: Lists differ: [] != ['/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py']
E       
E       Second list contains 1 additional elements.
E       First extra element 0:
E       '/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py'
E       
E       - []
E       + ['/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py']

tests/core/test_project_structure.py:230: AssertionError

Member

ashb commented Oct 29, 2020

@potiuk Selective tests (? or something else) caused CI to not detect failing tests:

https://github.com/apache/airflow/runs/1307780299

tests/core/test_project_structure.py sssssssssss                         [ 23%]

ashb added a commit to astronomer/airflow that referenced this pull request


          Rename example JDBC dag

19db64c

This was merged in apache#11472, but for some reason that didn't run tests so
the failure wasn't noticed

ashb mentioned this pull request

Rename example JDBC dag #11946

Merged

ashb added a commit that referenced this pull request


          Rename example JDBC dag (#11946)

a7ad204

This was merged in #11472, but for some reason that didn't run tests so
the failure wasn't noticed

kaxil mentioned this pull request

Fix failing docs build on Master #11951

Merged

kaxil added a commit that referenced this pull request


          Fix failing docs build on Master (#11951)

2fe2011

This was merged in #11472, but for some reason that didn't run tests so
the failure wasn't noticed.

a7ad204 fixed 1 error but the docs error was missed

Contributor Author

francescomucio commented Oct 29, 2020

Sorry for the mess, not sure what I should keep an eye on next time

michalmisiewicz pushed a commit to michalmisiewicz/airflow that referenced this pull request


          Add How-to guide for JDBC Operator (apache#11472)

2ff15ae

michalmisiewicz pushed a commit to michalmisiewicz/airflow that referenced this pull request


          Rename example JDBC dag (apache#11946)

8d51894

This was merged in apache#11472, but for some reason that didn't run tests so
the failure wasn't noticed

michalmisiewicz pushed a commit to michalmisiewicz/airflow that referenced this pull request


          Fix failing docs build on Master (apache#11951)

178155c

This was merged in apache#11472, but for some reason that didn't run tests so
the failure wasn't noticed.

a7ad204 fixed 1 error but the docs error was missed

szn pushed a commit to szn/airflow that referenced this pull request


          Add How-to guide for JDBC Operator (apache#11472)

4d878a4

szn pushed a commit to szn/airflow that referenced this pull request


          Rename example JDBC dag (apache#11946)

01a0947

This was merged in apache#11472, but for some reason that didn't run tests so
the failure wasn't noticed

szn pushed a commit to szn/airflow that referenced this pull request


          Fix failing docs build on Master (apache#11951)

45291f5

This was merged in apache#11472, but for some reason that didn't run tests so
the failure wasn't noticed.

a7ad204 fixed 1 error but the docs error was missed

eladkal mentioned this pull request

Add new datetime branch operator #11964

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment