Support for native import from Oracle (issue #26) #27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- PROP_IMPORT_FROM_ORA - PROP_ORA_CONNECTION_STRING
With the exception of rowid and urowid, the representation of all Oracle data types in Exasol is tested. All test cases are now run twice: once using a JDBC adapter connection and once using an IMPORT FROM ORA connection. Their results are tested for equivalence.
Reasons for slow test execution: invalid SCRIPT_OUTPUT_ADDRESS and blocking randomness source.
andrehacker
approved these changes
Jun 28, 2018
There was a problem hiding this comment.
Great contrib @hendrikcech ! I merge this now because a customer was asking for the feature and all my comments are cosmetically. You can work on the few issues in a separate pull request.
|
|
||
| **Mandatory Connection Specification:** | ||
| Either specify `CONNECTION_NAME` OR provide `CONNECTION_STRING`,String `USERNAME` and `PASSWORD`. |
There was a problem hiding this comment.
A visible new line before Either would be nice to make clear the this is a new subsection.
There was a problem hiding this comment.
The "String" word seems to be need needed.
| **Advanced Optional Properties:** | ||
|
|
||
| Parameter | Value | ||
| Property | Value | ||
| --------------------------- | ----------- | ||
| **IMPORT_FROM_EXA** | Only relevant if your data source is EXASOL. Either 'TRUE' or 'FALSE' (default). If true, IMPORT FROM EXA will be used for the pushdown instead of IMPORT FROM JDBC. You have to define EXA_CONNECTION_STRING if this property is true. |
There was a problem hiding this comment.
Would be cool if you could change all the SQL statements and property names to monospace, i.e. IMPORT FROM EXA/JDBC and EXA_CONNECTION_STRING here.
Please review at least the Adapter Properties section, I think there are many properties which we could convert to monospace. Then we have this finally consistent. You can make a separate pull request for this documentation fixes.
|
|
||
| * **Very slow execution of queries with SCRIPT_OUTPUT_ADDRESS**: If `SCRIPT_OUTPUT_ADDRESS` is set as explained in the [debugging section](#debugging), verify that a service is actually listening at that address. Otherwise, if Exasol can not establish a connection, repeated connection attempts can be the cause for slowdowns. | ||
| * **Very slow execution of queries**: Depending on which JDK version Exasol uses to execute Java user-defined functions, a blocking randomness source may be used by default. Especially cryptographic operations do not complete until the operating system has collected a sufficient amount of entropy. This problem seems to occur most often when Exasol is run in an isolated environment, e.g., a virtual machine or a container. A solution is to use a non-blocking randomness source. | ||
| To do so, log in to EXAOperation and shutdown the database. Append `-etlJdbcJavaEnv -Djava.security.egd=/dev/./urandom` to the "Extra Database Parameters" input field and power the database on again. |
There was a problem hiding this comment.
This will fix the slowness of the IMPORT FROM JDBC, however, there are also JDBC metadata calls directly from the Adapter script, which will be executed in a nested container environment.
If this is the case, the %env (or javaoptions, forgot the details) would need to be set in the adapter script.
Did you also do this to fix the slowness issues?
There was a problem hiding this comment.
I'm not so sure about this case. Before submitting this PR I did some quick tests with setting the Java options in both EXAOperations and the adapter script (using %env variables), setting them in only one of the two, and not setting them at all. Here are the results running the integration tests two times for each combination (y meaning "Java option set", n meaning "option not set"):
EXAOperation y, Adapter Script n: 06:32 min / 6:32 min
EXAOperation y, Adapter Script y: 06:30 min / 6:46 min
EXAOperation n, Adapter Script y: 29:47 min / 31:09 min
EXAOperation n, Adapter Script n: 14:35 min / 14:50 min
The results are pretty surprising (especially comparing the last two cases) but that's what I based my recommendation on.
Merged
hendrikcech
added a commit
to hendrikcech/virtual-schemas
that referenced
this pull request
Jul 17, 2018
hendrikcech
added a commit
to hendrikcech/virtual-schemas
that referenced
this pull request
Jul 17, 2018
redcatbear
pushed a commit
that referenced
this pull request
Sep 11, 2018
redcatbear
added a commit
that referenced
this pull request
Oct 2, 2018
* Rename EXASOL to Exasol * Update supported-dialects.md * SQL generation to use SQL standard "<>" for not-equal predicate * Support for native import from Oracle (issue #26) (#27) * Increment to version 1.0.1 for releasing * Increment version to 6.0.2-SNAPSHOT for next development iteration * Readme fixes (#29) * Display SQL statements, etc. in monospace and clean up syntax * Add newline after heading * Add script to run Exasol integration tests on Travis CI (#28) A new shell script `run_integration_tests.sh` executes the integration tests as defined in `integration-test-travis.yaml`. It uses the exasol/docker-db image to spin up an Exasol instance and execute the Exasol dialect integration tests. Travis CI automatically executes the test script for each new commit. * refactor(PMI-69): Turned hard-coded SQL dialect list into registry. * feat(PMI-16): Dialect registry now uses scanning. * fix(PMI-16): Explicitly set report output encoding to avoid warning. * PMI-16: Added "local" directory to .gitignore * PMI-16: Got new SQL dialect registry running and improved logging. * PMI-16: Got properties-controlled dialects regsitry running. * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * Sybase Virtual Schema (#32) * Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects * Pmi 73 take over sybase from hendrik (#33) * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements. * PMI-74: Used "mktemp". Improved configurability and readability. * PMI-74: Parameterized more hard-coded paths. * PMI-74: Change temp dir to rooted temp dir. * PMI-74: Improved readability by splitting file into functions. * PIM-74: Travis badge now points to default branch. * PMI-75: Adapted to new SQL dialect registry. * Added local and Scripts to .gitignore * Added .dbeaver* to .gitignore * PMI-75: Cleaned up and split documentation. * PMI-75: Fixed integration tests. Fixed database preparation scripts * PMI-75: Refactored IntegrationTestSetup.java for better readability and to reduce static methods to a minimum. * PMI-75: Got remote logging running. * PMI-75: Improved log message formatting. Added unit tests for custom formatter. * PMI-75: Set default log level explicitly. * PMI-75: Added fallback to STDOUT in case socket output stream is not available. * PMI-75: Improved documentation. * PMI-75: Removed distracting introduction.
andrehacker
added a commit
that referenced
this pull request
Oct 4, 2018
* Sybase Virtual Schema (#32) * Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects * Pmi 73 take over sybase from hendrik (#33) * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements. * Pmi 74 improve travis ci build (#35) * PMI-74: Used "mktemp". Improved configurability and readability. * PMI-74: Parameterized more hard-coded paths. * PMI-74: Change temp dir to rooted temp dir. * PMI-74: Improved readability by splitting file into functions. * PIM-74: Travis badge now points to default branch. * Update README.md * Added DBeaver files to ".gitignore" * Feature/pmi 69 invert dialect dependencies (#31) * refactor(PMI-69): Turned hard-coded SQL dialect list into registry. * feat(PMI-16): Dialect registry now uses scanning. * fix(PMI-16): Explicitly set report output encoding to avoid warning. * PMI-16: Added "local" directory to .gitignore * PMI-16: Got new SQL dialect registry running and improved logging. * PMI-16: Got properties-controlled dialects regsitry running. * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-69: Fixed review findings of Andre Hacker. * PMI-69: Removed superfluous "@OverRide". * Pmi 75 sybase integration test (#36) * Rename EXASOL to Exasol * Update supported-dialects.md * SQL generation to use SQL standard "<>" for not-equal predicate * Support for native import from Oracle (issue #26) (#27) * Increment to version 1.0.1 for releasing * Increment version to 6.0.2-SNAPSHOT for next development iteration * Readme fixes (#29) * Display SQL statements, etc. in monospace and clean up syntax * Add newline after heading * Add script to run Exasol integration tests on Travis CI (#28) A new shell script `run_integration_tests.sh` executes the integration tests as defined in `integration-test-travis.yaml`. It uses the exasol/docker-db image to spin up an Exasol instance and execute the Exasol dialect integration tests. Travis CI automatically executes the test script for each new commit. * refactor(PMI-69): Turned hard-coded SQL dialect list into registry. * feat(PMI-16): Dialect registry now uses scanning. * fix(PMI-16): Explicitly set report output encoding to avoid warning. * PMI-16: Added "local" directory to .gitignore * PMI-16: Got new SQL dialect registry running and improved logging. * PMI-16: Got properties-controlled dialects regsitry running. * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * Sybase Virtual Schema (#32) * Use SqlServer dialect as starting point * Add sybase dialect and override for ORDER BY * Setup Sybase integration tests: includes order by tests * Add WHERE test * Date/time types: handle conversion and add tests * Add sybase integer datatype tests * Add a whole bunch of data type tests * Add binary, varbinary, image and bit tests * Add Sybase to the supported dialects * Pmi 73 take over sybase from hendrik (#33) * PMI-16: Set fixed database version for CI build. Cleaned up integration test script. * PMI-16: Fixed quoting. * PMI-16: Corrected docker image version. * PMI-76: Cherry-picked integration test script improvements. * PMI-74: Used "mktemp". Improved configurability and readability. * PMI-74: Parameterized more hard-coded paths. * PMI-74: Change temp dir to rooted temp dir. * PMI-74: Improved readability by splitting file into functions. * PIM-74: Travis badge now points to default branch. * PMI-75: Adapted to new SQL dialect registry. * Added local and Scripts to .gitignore * Added .dbeaver* to .gitignore * PMI-75: Cleaned up and split documentation. * PMI-75: Fixed integration tests. Fixed database preparation scripts * PMI-75: Refactored IntegrationTestSetup.java for better readability and to reduce static methods to a minimum. * PMI-75: Got remote logging running. * PMI-75: Improved log message formatting. Added unit tests for custom formatter. * PMI-75: Set default log level explicitly. * PMI-75: Added fallback to STDOUT in case socket output stream is not available. * PMI-75: Improved documentation. * PMI-75: Removed distracting introduction. * Feature/pmi 91 update version to 1.1.0 (#37) * Deleted `increment-version.sh` script * Created `version.sh` script that has a verification and an updater mode * Introduced `product.version` property in master `pom.xml` * Used that property in all child-POM files * Used `version.sh` to update documentation * Added `version.sh verify` as build breaker in `.travis.yml`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request addresses the feature described by issue #26. Changes include:
IMPORT_FROM_ORAandORA_CONNECTION_NAMEare added.JdbcAdapter.handlePushdownRequest()is extended to generateIMPORT FROM ORAqueries.OracleSqlDialectIT). All tests are run twice; once over a JDBC and once over an IMPORT FROM ORA connection.oracle-testdata.sqlis included which prepares an Oracle table that is used by the Oracle integration tests.