Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add MSSQL support #1538

Merged
merged 11 commits into from
Jan 25, 2023
Merged

Add MSSQL support #1538

merged 11 commits into from
Jan 25, 2023

Conversation

phanikumv
Copy link
Collaborator

@phanikumv phanikumv commented Jan 6, 2023 •
edited
Loading

Description

What is the current behavior?

astro-sdk operators currently do not work with mssql

closes: #1109

What is the new behavior?

  • Add mssql support

Does this introduce a breaking change?

No

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README / documentation, if necessary

@codecov
Copy link

codecov bot commented Jan 6, 2023 •
edited
Loading

Codecov Report

Base: 97.72% // Head: 97.72% // No change to project coverage 👍

Coverage data is based on head (e3560d4) compared to base (21dcaca).
Patch has no changes to coverable lines.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #1538   +/-   ##
=======================================
  Coverage   97.72%   97.72%           
=======================================
  Files          21       21           
  Lines         835      835           
=======================================
  Hits          816      816           
  Misses         19       19

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@phanikumv phanikumv force-pushed the add-mssql branch 10 times, most recently from 8b7794b to 404ab20 Compare January 10, 2023 06:46
@phanikumv phanikumv marked this pull request as ready for review January 10, 2023 09:35
sunank200
sunank200 reviewed Jan 10, 2023
View reviewed changes
Copy link
Contributor

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments but i think we should also add few integration test as part of example DAG for mysql as well. WDYT?

python-sdk/src/astro/databases/base.py Outdated Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Outdated Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Outdated Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Show resolved Hide resolved
python-sdk/tests_integration/sql/operators/test_load_file.py Show resolved Hide resolved
python-sdk/tests_integration/sql/operators/test_load_file.py Outdated Show resolved Hide resolved
@phanikumv phanikumv force-pushed the add-mssql branch 2 times, most recently from ba8a335 to b2366e5 Compare January 11, 2023 06:18
@phanikumv phanikumv force-pushed the add-mssql branch 2 times, most recently from c91e941 to f2f5b75 Compare January 19, 2023 19:30
@kaxil kaxil changed the title Add mssql support Add MSSQL support Jan 19, 2023
feluelle
feluelle reviewed Jan 23, 2023
View reviewed changes
python-sdk/noxfile.py Outdated Show resolved Hide resolved
python-sdk/src/astro/databases/mssql.py Outdated Show resolved Hide resolved
python-sdk/tests_integration/databases/test_all_databases.py Show resolved Hide resolved
@phanikumv phanikumv force-pushed the add-mssql branch 3 times, most recently from fff847c to 16b3a4d Compare January 24, 2023 07:31
pankajastro
pankajastro approved these changes Jan 25, 2023
View reviewed changes
python-sdk/example_dags/example_load_file.py Show resolved Hide resolved
python-sdk/example_dags/example_mssql_transform.py Outdated Show resolved Hide resolved
python-sdk/example_dags/example_mssql_transform.py Outdated Show resolved Hide resolved
python-sdk/example_dags/example_mssql_transform.py Outdated Show resolved Hide resolved
python-sdk/example_dags/example_mssql_transform.py Outdated Show resolved Hide resolved
python-sdk/noxfile.py Show resolved Hide resolved
python-sdk/noxfile.py Outdated Show resolved Hide resolved
@phanikumv
Copy link
Collaborator Author

Added comments but i think we should also add few integration test as part of example DAG for mysql as well. WDYT?

This is completed

@phanikumv
Copy link
Collaborator Author

phanikumv commented Jan 25, 2023 •
edited
Loading

Overall the PR seems in much better shape now. Added few comments. Also following things would be required:

  • documentation changes for mssql
  • Example DAG in loadfile and export file

Added documentation and example DAG

@phanikumv phanikumv merged commit 68cc8a4 into main Jan 25, 2023
@phanikumv phanikumv deleted the add-mssql branch January 25, 2023 08:04
@phanikumv phanikumv mentioned this pull request Jan 31, 2023
Merged
phanikumv added a commit that referenced this pull request Feb 1, 2023
@phanikumv
Add pymssql dependent libraries to dev Dockerfile (#1690)
7f85f4f 
# Description
## What is the current behavior?
After including the SQL server support for astro-sdk in #1538 , we are
unable to spin up the dev docker container using astro-runtime 7.2, with
the below error.This error is due to the libraries required for
`pymssql` successful installation on the container missing in the
`dev/Dockerfile`


```
#0 41.91   Building wheel for pymssql (pyproject.toml): started
#0 43.11   Building wheel for pymssql (pyproject.toml): finished with status 'error'
#0 43.11   error: subprocess-exited-with-error
#0 43.11   
#0 43.11   × Building wheel for pymssql (pyproject.toml) did not run successfully.
#0 43.11   │ exit code: 1
#0 43.11   ╰─> [25 lines of output]
#0 43.11       setup.py: platform.system() => Linux
#0 43.11       setup.py: platform.architecture() => ('64bit', '')
#0 43.11       setup.py: platform.libc_ver() => ('glibc', '2.31')
#0 43.11       setup.py: include_dirs => []
#0 43.11       setup.py: library_dirs => []
#0 43.11       running bdist_wheel
#0 43.11       running build
#0 43.11       running build_py
#0 43.11       creating build
#0 43.11       creating build/lib.linux-aarch64-3.9
#0 43.11       creating build/lib.linux-aarch64-3.9/pymssql
#0 43.11       copying src/pymssql/__init__.py -> build/lib.linux-aarch64-3.9/pymssql
#0 43.11       running build_ext
#0 43.11       cythoning src/pymssql/_mssql.pyx to src/pymssql/_mssql.c
#0 43.11       cythoning src/pymssql/_pymssql.pyx to src/pymssql/_pymssql.c
#0 43.11       building 'pymssql._mssql' extension
#0 43.11       creating build/temp.linux-aarch64-3.9
#0 43.11       creating build/temp.linux-aarch64-3.9/src
#0 43.11       creating build/temp.linux-aarch64-3.9/src/pymssql
#0 43.11       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/pymssql/_mssql.c -o build/temp.linux-aarch64-3.9/src/pymssql/_mssql.o -DMSDBLIB
#0 43.11       src/pymssql/_mssql.c:747:10: fatal error: sqlfront.h: No such file or directory
#0 43.11         747 | #include "sqlfront.h"
#0 43.11             |          ^~~~~~~~~~~~
#0 43.11       compilation terminated.
#0 43.11       error: command '/usr/bin/gcc' failed with exit code 1
#0 43.11       [end of output]

```

<!-- Please describe the current behavior that you are modifying. -->


<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Include the missing libraries for successful `pymssql` installation in
the `dev/Dockerfile`


## Does this introduce a breaking change?
No
utkarsharma2 pushed a commit that referenced this pull request Feb 2, 2023
@phanikumv @utkarsharma2
Add pymssql dependent libraries to dev Dockerfile (#1690)
6020f0f 
# Description
## What is the current behavior?
After including the SQL server support for astro-sdk in #1538 , we are
unable to spin up the dev docker container using astro-runtime 7.2, with
the below error.This error is due to the libraries required for
`pymssql` successful installation on the container missing in the
`dev/Dockerfile`


```
#0 41.91   Building wheel for pymssql (pyproject.toml): started
#0 43.11   Building wheel for pymssql (pyproject.toml): finished with status 'error'
#0 43.11   error: subprocess-exited-with-error
#0 43.11   
#0 43.11   × Building wheel for pymssql (pyproject.toml) did not run successfully.
#0 43.11   │ exit code: 1
#0 43.11   ╰─> [25 lines of output]
#0 43.11       setup.py: platform.system() => Linux
#0 43.11       setup.py: platform.architecture() => ('64bit', '')
#0 43.11       setup.py: platform.libc_ver() => ('glibc', '2.31')
#0 43.11       setup.py: include_dirs => []
#0 43.11       setup.py: library_dirs => []
#0 43.11       running bdist_wheel
#0 43.11       running build
#0 43.11       running build_py
#0 43.11       creating build
#0 43.11       creating build/lib.linux-aarch64-3.9
#0 43.11       creating build/lib.linux-aarch64-3.9/pymssql
#0 43.11       copying src/pymssql/__init__.py -> build/lib.linux-aarch64-3.9/pymssql
#0 43.11       running build_ext
#0 43.11       cythoning src/pymssql/_mssql.pyx to src/pymssql/_mssql.c
#0 43.11       cythoning src/pymssql/_pymssql.pyx to src/pymssql/_pymssql.c
#0 43.11       building 'pymssql._mssql' extension
#0 43.11       creating build/temp.linux-aarch64-3.9
#0 43.11       creating build/temp.linux-aarch64-3.9/src
#0 43.11       creating build/temp.linux-aarch64-3.9/src/pymssql
#0 43.11       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/pymssql/_mssql.c -o build/temp.linux-aarch64-3.9/src/pymssql/_mssql.o -DMSDBLIB
#0 43.11       src/pymssql/_mssql.c:747:10: fatal error: sqlfront.h: No such file or directory
#0 43.11         747 | #include "sqlfront.h"
#0 43.11             |          ^~~~~~~~~~~~
#0 43.11       compilation terminated.
#0 43.11       error: command '/usr/bin/gcc' failed with exit code 1
#0 43.11       [end of output]

```

<!-- Please describe the current behavior that you are modifying. -->


<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Include the missing libraries for successful `pymssql` installation in
the `dev/Dockerfile`


## Does this introduce a breaking change?
No
utkarsharma2 pushed a commit that referenced this pull request Feb 8, 2023
@phanikumv @utkarsharma2
Add MSSQL support (#1538)
98342f9 
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
astro-sdk operators currently do not work with mssql


closes: #1109 


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Add mssql support


## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary
utkarsharma2 pushed a commit that referenced this pull request Feb 8, 2023
@phanikumv @utkarsharma2
Add pymssql dependent libraries to dev Dockerfile (#1690)
f182aa7 
# Description
## What is the current behavior?
After including the SQL server support for astro-sdk in #1538 , we are
unable to spin up the dev docker container using astro-runtime 7.2, with
the below error.This error is due to the libraries required for
`pymssql` successful installation on the container missing in the
`dev/Dockerfile`


```
#0 41.91   Building wheel for pymssql (pyproject.toml): started
#0 43.11   Building wheel for pymssql (pyproject.toml): finished with status 'error'
#0 43.11   error: subprocess-exited-with-error
#0 43.11   
#0 43.11   × Building wheel for pymssql (pyproject.toml) did not run successfully.
#0 43.11   │ exit code: 1
#0 43.11   ╰─> [25 lines of output]
#0 43.11       setup.py: platform.system() => Linux
#0 43.11       setup.py: platform.architecture() => ('64bit', '')
#0 43.11       setup.py: platform.libc_ver() => ('glibc', '2.31')
#0 43.11       setup.py: include_dirs => []
#0 43.11       setup.py: library_dirs => []
#0 43.11       running bdist_wheel
#0 43.11       running build
#0 43.11       running build_py
#0 43.11       creating build
#0 43.11       creating build/lib.linux-aarch64-3.9
#0 43.11       creating build/lib.linux-aarch64-3.9/pymssql
#0 43.11       copying src/pymssql/__init__.py -> build/lib.linux-aarch64-3.9/pymssql
#0 43.11       running build_ext
#0 43.11       cythoning src/pymssql/_mssql.pyx to src/pymssql/_mssql.c
#0 43.11       cythoning src/pymssql/_pymssql.pyx to src/pymssql/_pymssql.c
#0 43.11       building 'pymssql._mssql' extension
#0 43.11       creating build/temp.linux-aarch64-3.9
#0 43.11       creating build/temp.linux-aarch64-3.9/src
#0 43.11       creating build/temp.linux-aarch64-3.9/src/pymssql
#0 43.11       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/pymssql/_mssql.c -o build/temp.linux-aarch64-3.9/src/pymssql/_mssql.o -DMSDBLIB
#0 43.11       src/pymssql/_mssql.c:747:10: fatal error: sqlfront.h: No such file or directory
#0 43.11         747 | #include "sqlfront.h"
#0 43.11             |          ^~~~~~~~~~~~
#0 43.11       compilation terminated.
#0 43.11       error: command '/usr/bin/gcc' failed with exit code 1
#0 43.11       [end of output]

```

<!-- Please describe the current behavior that you are modifying. -->


<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Include the missing libraries for successful `pymssql` installation in
the `dev/Dockerfile`


## Does this introduce a breaking change?
No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil left review comments

@feluelle feluelle feluelle left review comments

@pankajastro pankajastro pankajastro approved these changes

@dimberman dimberman Awaiting requested review from dimberman dimberman is a code owner

@pankajkoti pankajkoti Awaiting requested review from pankajkoti

@tatiana tatiana Awaiting requested review from tatiana tatiana is a code owner

@utkarsharma2 utkarsharma2 Awaiting requested review from utkarsharma2 utkarsharma2 is a code owner

@sunank200 sunank200 Awaiting requested review from sunank200 sunank200 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SQL Server Support
7 participants
@phanikumv @tatiana @sunank200 @kaxil @utkarsharma2 @feluelle @pankajastro