Pmi 75 sybase integration test #36

redcatbear · 2018-09-17T14:08:27Z

Summary:

Switched Logging from System.out to java.util.logging
Created customer formatter com.exasol.logging.CompactFormatter
Local logging configurable via standard logging.properties
Remote logging configurable in CREATE VIRTUAL SCHEMA statement
Refactored and fixed output stream connection to socket
Updated and cleaned-up documentation
One document per dialect
Improved test coverage
Split SQL scripts for Sybase into phases for faster testing cycles

* Display SQL statements, etc. in monospace and clean up syntax * Add newline after heading

A new shell script `run_integration_tests.sh` executes the integration tests as defined in `integration-test-travis.yaml`. It uses the exasol/docker-db image to spin up an Exasol instance and execute the Exasol dialect integration tests. Travis CI automatically executes the test script for each new commit.

…on test script.

* Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects

* PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements.

…gration_test

…_Sybase_integration_test Conflicts: .gitignore jdbc-adapter/integration-test-data/run_integration_tests.sh jdbc-adapter/virtualschema-jdbc-adapter/src/main/java/com/exasol/adapter/dialects/SqlDialects.java jdbc-adapter/virtualschema-jdbc-adapter/src/main/java/com/exasol/adapter/jdbc/JdbcAdapter.java

…nd to reduce static methods to a minimum.

…formatter.

andrehacker

Great PR again improving the consistency of the documentation and more. See my comments regarding the list of dialects. BTW, I like the change to have a separate subpage for each dialect.

andrehacker · 2018-10-01T18:58:36Z

jdbc-adapter/README.md

-* PostgreSQL
+The JDBC adapter for virtual schemas allows you to connect to JDBC data sources like Hive, Oracle, Teradata, Exasol or any other data source supporting JDBC. It uses the well proven ```IMPORT FROM JDBC``` Exasol statement behind the scenes to obtain the requested data, when running a query on a virtual table. The JDBC adapter also serves as the reference adapter for the Exasol virtual schema framework.
+
+Check the [SQL dialect list](doc/supported_sql_dialects.md) to learn which SQL dialects the JDBC adapter currently supports


I see the point that it is not good to maintain two separate lists. However I liked how it was before where you could see at the beginning of the main JDBC Adapter readme which dialects are supported. This is one of the most important things to show, and after this PR it would be relatively hidden on a subpage (where you have to scroll down first). Also I am not yet sure if we need a subpage supported_sql_dialects.md, since many subpages can make things rather complex.
So I think about the following:

Add the list with the supported dialects again to the main README, with links to the relevant pages. Maybe with a separate subsection (still on top) so that we can link to it

in the getting started guide we can link to the section in the README

There are more solutions for sure, let's discuss.

andrehacker · 2018-10-01T19:03:03Z

jdbc-adapter/README.md


-If you are interested in a introduction to virtual schemas please refer to the EXASOL user manual. You can find it in the [download area of the EXASOL user portal](https://www.exasol.com/portal/display/DOWNLOAD/6.0).
+In addition to the aforementioned dialects there is the so called `GENERIC` dialect, which is designed to work with any JDBC driver. It derives the SQL dialect from the JDBC driver metadata. However, it does not support any capabilities and might fail if the data source has special syntax or data types, so it should only be used for evaluation purposes.


I like the sentence, but it makes more sense if we directly mention the dialects above

andrehacker · 2018-10-01T19:04:13Z

jdbc-adapter/README.md


+If you are interested in a introduction to virtual schemas please refer to the Exasol user manual. You can find it in the [download area of the Exasol user portal](https://www.exasol.com/portal/display/DOWNLOAD/6.0).


Improvement proposal: Here is a better link: https://www.exasol.com/portal/display/DOC/Database+User+Manual. You could make a link out of Exasol user manual. (not really part of this PR - nice to have)

Exchanged link.

andrehacker · 2018-10-01T19:08:08Z

jdbc-adapter/doc/supported_sql_dialects.md

@@ -0,0 +1,26 @@
+# Supported Dialects


See my first comment about this new subpage

andrehacker · 2018-10-01T19:23:37Z

...rtualschema-jdbc-adapter/src/test/java/com/exasol/adapter/dialects/IntegrationTestSetup.java

+ *
+ * <p>
+ * This does not deploy the latest JAR, because the distribution Maven module is
+ * and and build after this module. Right now you need to do something like


and and typo.

Right now I wonder if it is not possible to do it all in one Maven call. Maybe the integration tests can be moved into the pom.xml on highest level and this will be executed at the end? Not sure, ... Anyway, such change is not part of this PR.

The integration tests need to be redone from the ground on.
They need to automatically create a clean environment for each test, we need cleaner and most importantly more compact configuration and uniform testing across all dialects. While we are at it we also need to eliminate code duplication.

That is a separate PR for sure.

…gration_test Conflicts: .gitignore jdbc-adapter/README.md jdbc-adapter/doc/deploy-adapter.md jdbc-adapter/doc/develop-dialect.md jdbc-adapter/doc/supported-dialects.md jdbc-adapter/integration-test-data/run_integration_tests.sh jdbc-adapter/virtualschema-jdbc-adapter/src/main/java/com/exasol/adapter/dialects/impl/SybaseSqlDialect.java jdbc-adapter/virtualschema-jdbc-adapter/src/main/java/com/exasol/adapter/jdbc/JdbcAdapter.java jdbc-adapter/virtualschema-jdbc-adapter/src/main/java/com/exasol/adapter/jdbc/JdbcMetadataReader.java jdbc-adapter/virtualschema-jdbc-adapter/src/main/resources/sql_dialects.properties jdbc-adapter/virtualschema-jdbc-adapter/src/test/java/com/exasol/adapter/dialects/AbstractIntegrationTest.java jdbc-adapter/virtualschema-jdbc-adapter/src/test/java/com/exasol/adapter/dialects/IntegrationTestSetup.java jdbc-adapter/virtualschema-jdbc-adapter/src/test/java/com/exasol/adapter/dialects/impl/SybaseSqlDialectIT.java

…available.

redcatbear · 2018-10-02T09:09:41Z

@andrehacker: Implemented your requested changes. Please check again.

redcatbear · 2018-10-02T10:06:57Z

Accidentally hit the "close" button. Reopened.

@OverRide

* Sybase Virtual Schema (#32) * Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects * Pmi 73 take over sybase from hendrik (#33) * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements. * Pmi 74 improve travis ci build (#35) * PMI-74: Used "mktemp". Improved configurability and readability. * PMI-74: Parameterized more hard-coded paths. * PMI-74: Change temp dir to rooted temp dir. * PMI-74: Improved readability by splitting file into functions. * PIM-74: Travis badge now points to default branch. * Update README.md * Added DBeaver files to ".gitignore" * Feature/pmi 69 invert dialect dependencies (#31) * refactor(PMI-69): Turned hard-coded SQL dialect list into registry. * feat(PMI-16): Dialect registry now uses scanning. * fix(PMI-16): Explicitly set report output encoding to avoid warning. * PMI-16: Added "local" directory to .gitignore * PMI-16: Got new SQL dialect registry running and improved logging. * PMI-16: Got properties-controlled dialects regsitry running. * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-69: Fixed review findings of Andre Hacker. * PMI-69: Removed superfluous "@OverRide". * Pmi 75 sybase integration test (#36) * Rename EXASOL to Exasol * Update supported-dialects.md * SQL generation to use SQL standard "<>" for not-equal predicate * Support for native import from Oracle (issue #26) (#27) * Increment to version 1.0.1 for releasing * Increment version to 6.0.2-SNAPSHOT for next development iteration * Readme fixes (#29) * Display SQL statements, etc. in monospace and clean up syntax * Add newline after heading * Add script to run Exasol integration tests on Travis CI (#28) A new shell script `run_integration_tests.sh` executes the integration tests as defined in `integration-test-travis.yaml`. It uses the exasol/docker-db image to spin up an Exasol instance and execute the Exasol dialect integration tests. Travis CI automatically executes the test script for each new commit. * refactor(PMI-69): Turned hard-coded SQL dialect list into registry. * feat(PMI-16): Dialect registry now uses scanning. * fix(PMI-16): Explicitly set report output encoding to avoid warning. * PMI-16: Added "local" directory to .gitignore * PMI-16: Got new SQL dialect registry running and improved logging. * PMI-16: Got properties-controlled dialects regsitry running. * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * Sybase Virtual Schema (#32) * Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects * Pmi 73 take over sybase from hendrik (#33) * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements. * PMI-74: Used "mktemp". Improved configurability and readability. * PMI-74: Parameterized more hard-coded paths. * PMI-74: Change temp dir to rooted temp dir. * PMI-74: Improved readability by splitting file into functions. * PIM-74: Travis badge now points to default branch. * PMI-75: Adapted to new SQL dialect registry. * Added local and Scripts to .gitignore * Added .dbeaver* to .gitignore * PMI-75: Cleaned up and split documentation. * PMI-75: Fixed integration tests. Fixed database preparation scripts * PMI-75: Refactored IntegrationTestSetup.java for better readability and to reduce static methods to a minimum. * PMI-75: Got remote logging running. * PMI-75: Improved log message formatting. Added unit tests for custom formatter. * PMI-75: Set default log level explicitly. * PMI-75: Added fallback to STDOUT in case socket output stream is not available. * PMI-75: Improved documentation. * PMI-75: Removed distracting introduction. * Feature/pmi 91 update version to 1.1.0 (#37) * Deleted `increment-version.sh` script * Created `version.sh` script that has a verification and an updater mode * Introduced `product.version` property in master `pom.xml` * Used that property in all child-POM files * Used `version.sh` to update documentation * Added `version.sh verify` as build breaker in `.travis.yml`

andrehacker and others added 30 commits December 1, 2017 21:52

Rename EXASOL to Exasol

3d9d9e9

Update supported-dialects.md

7eda260

SQL generation to use SQL standard "<>" for not-equal predicate

682712f

Support for native import from Oracle (issue #26) (#27)

5865e71

Increment to version 1.0.1 for releasing

9a37601

Increment version to 6.0.2-SNAPSHOT for next development iteration

b54dbf5

Readme fixes (#29)

4a24ae2

* Display SQL statements, etc. in monospace and clean up syntax * Add newline after heading

refactor(PMI-69): Turned hard-coded SQL dialect list into registry.

1bf7c34

feat(PMI-16): Dialect registry now uses scanning.

48cf825

fix(PMI-16): Explicitly set report output encoding to avoid warning.

f38a03f

PMI-16: Added "local" directory to .gitignore

ef99f14

PMI-16: Got new SQL dialect registry running and improved logging.

6daded4

PMI-16: Got properties-controlled dialects regsitry running.

94285e3

PMI-16: Set fixed database version for CI build. Cleaned up integrati…

40ded50

…on test script.

PMI-16: Fixed quoting.

3759e6d

PMI-16: Corrected docker image version.

42214c9

Pmi 73 take over sybase from hendrik (#33)

d72a7f4

* PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements.

PMI-74: Used "mktemp". Improved configurability and readability.

01023d7

PMI-74: Parameterized more hard-coded paths.

eee6418

PMI-74: Change temp dir to rooted temp dir.

4ddd103

PMI-74: Improved readability by splitting file into functions.

9883a27

PIM-74: Travis badge now points to default branch.

95ca464

PMI-75: Adapted to new SQL dialect registry.

c4fd29e

Added local and Scripts to .gitignore

2d4af07

Merge branch 'PMI-74_Improve_Travis_CI_build' into PMI-75_Sybase_inte…

d71e339

…gration_test

Added .dbeaver* to .gitignore

733b09d

PMI-75: Cleaned up and split documentation.

fa7d0e3

redcatbear added 4 commits September 12, 2018 14:25

PMI-75: Fixed integration tests. Fixed database preparation scripts

a5f965d

PMI-75: Refactored IntegrationTestSetup.java for better readability a…

a400ba4

…nd to reduce static methods to a minimum.

PMI-75: Got remote logging running.

e185bb9

PMI-75: Improved log message formatting. Added unit tests for custom …

8b9b621

…formatter.

andrehacker suggested changes Oct 1, 2018

View reviewed changes

redcatbear added 4 commits October 2, 2018 09:11

PMI-75: Set default log level explicitly.

74f3ca6

PMI-75: Added fallback to STDOUT in case socket output stream is not …

9db55ed

…available.

PMI-75: Improved documentation.

995611c

redcatbear closed this Oct 2, 2018

redcatbear reopened this Oct 2, 2018

PMI-75: Removed distracting introduction.

8447610

andrehacker approved these changes Oct 2, 2018

View reviewed changes

redcatbear merged commit b4634a0 into develop Oct 2, 2018

redcatbear deleted the PMI-75_Sybase_integration_test branch October 2, 2018 11:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pmi 75 sybase integration test #36

Pmi 75 sybase integration test #36

redcatbear commented Sep 17, 2018

andrehacker left a comment •

edited

Loading

andrehacker Oct 1, 2018

redcatbear Oct 2, 2018

andrehacker Oct 1, 2018

redcatbear Oct 2, 2018

andrehacker Oct 1, 2018

redcatbear Oct 2, 2018

andrehacker Oct 1, 2018

redcatbear Oct 2, 2018

andrehacker Oct 1, 2018

redcatbear Oct 2, 2018

redcatbear commented Oct 2, 2018

redcatbear commented Oct 2, 2018


		If you are interested in a introduction to virtual schemas please refer to the EXASOL user manual. You can find it in the [download area of the EXASOL user portal](https://www.exasol.com/portal/display/DOWNLOAD/6.0).
		In addition to the aforementioned dialects there is the so called `GENERIC` dialect, which is designed to work with any JDBC driver. It derives the SQL dialect from the JDBC driver metadata. However, it does not support any capabilities and might fail if the data source has special syntax or data types, so it should only be used for evaluation purposes.

Pmi 75 sybase integration test #36

Pmi 75 sybase integration test #36

Conversation

redcatbear commented Sep 17, 2018

andrehacker left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redcatbear commented Oct 2, 2018

redcatbear commented Oct 2, 2018

andrehacker left a comment •

edited

Loading