Skip to content

PLUGIN-1535 : Oracle decimal type Backward Compatibility Fix #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MrRahulSharma merged 1 commit into from
Mar 14, 2023
Merged

PLUGIN-1535 : Oracle decimal type Backward Compatibility Fix #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 32 additions & 31 deletions oracle-plugin/docs/Oracle-batchsink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,37 +60,38 @@ will be passed to the JDBC driver as connection arguments for JDBC drivers that
Data Types Mapping
----------

| Oracle Data Type | CDAP Schema Data Type | Comment |
| ------------------------------ | --------------------- | ------------------------------------------------------ |
| VARCHAR2 | string | |
| NVARCHAR2 | string | |
| VARCHAR | string | |
| NUMBER | string | For NUMBER types defined without a precision and scale |
| NUMBER | decimal | For NUMBER types with a defined precision and scale |
| FLOAT | double | |
| LONG | string | |
| DATE | timestamp | |
| BINARY_FLOAT | float | |
| BINARY_DOUBLE | double | |
| TIMESTAMP | timestamp | |
| TIMESTAMP WITH TIME ZONE | string | Timestamp string in the following format: |
| | | "2019-07-15 15:57:46.65 GMT" |
| TIMESTAMP WITH LOCAL TIME ZONE | timestamp | |
| INTERVAL YEAR TO MONTH | string | Oracle's 'INTERVAL YEAR TO MONTH' literal in the |
| | | standard format: "year[-month]" |
| INTERVAL DAY TO SECOND | string | Oracle's 'INTERVAL DAY TO SECOND' literal in the |
| | | standard format: |
| | | "[day] [hour][:minutes][:seconds[.milliseconds]" |
| RAW | bytes | |
| LONG RAW | bytes | |
| ROWID | string | |
| UROWID | string | |
| CHAR | string | |
| NCHAR | string | |
| CLOB | string | |
| NCLOB | string | |
| BLOB | bytes | |
| BFILE | | BFILE data type is not supported for the sink |
| Oracle Data Type | CDAP Schema Data Type | Comment |
| ------------------------------ | --------------------- | -----------------------------------------------------------|
| VARCHAR2 | string | |
| NVARCHAR2 | string | |
| VARCHAR | string | |
| NUMBER | string | For NUMBER types defined without a precision and scale. |
| | | Users can manually set output schema to map it to Decimal. |
| NUMBER | decimal | For NUMBER types defined with a precision and scale. |
| FLOAT | double | |
| LONG | string | |
| DATE | timestamp | |
| BINARY_FLOAT | float | |
| BINARY_DOUBLE | double | |
| TIMESTAMP | timestamp | |
| TIMESTAMP WITH TIME ZONE | string | Timestamp string in the following format: |
| | | "2019-07-15 15:57:46.65 GMT" |
| TIMESTAMP WITH LOCAL TIME ZONE | timestamp | |
| INTERVAL YEAR TO MONTH | string | Oracle's 'INTERVAL YEAR TO MONTH' literal in the |
| | | standard format: "year[-month]" |
| INTERVAL DAY TO SECOND | string | Oracle's 'INTERVAL DAY TO SECOND' literal in the |
| | | standard format: |
| | | "[day] [hour][:minutes][:seconds[.milliseconds]" |
| RAW | bytes | |
| LONG RAW | bytes | |
| ROWID | string | |
| UROWID | string | |
| CHAR | string | |
| NCHAR | string | |
| CLOB | string | |
| NCLOB | string | |
| BLOB | bytes | |
| BFILE | | BFILE data type is not supported for the sink |


Example
Expand Down
60 changes: 31 additions & 29 deletions oracle-plugin/docs/Oracle-batchsource.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,35 +74,37 @@ with the tradeoff of higher memory usage.
Data Types Mapping
----------

| Oracle Data Type | CDAP Schema Data Type | Comment |
| ------------------------------ | --------------------- | ------------------------------------------------------ |
| VARCHAR2 | string | |
| NVARCHAR2 | string | |
| VARCHAR | string | |
| NUMBER | decimal | |
| FLOAT | double | |
| LONG | string | |
| DATE | timestamp | |
| BINARY_FLOAT | float | |
| BINARY_DOUBLE | double | |
| TIMESTAMP | timestamp | |
| TIMESTAMP WITH TIME ZONE | string | |
| TIMESTAMP WITH LOCAL TIME ZONE | timestamp | |
| INTERVAL YEAR TO MONTH | string | |
| INTERVAL DAY TO SECOND | string | |
| RAW | bytes | |
| LONG RAW | bytes | |
| ROWID | string | |
| UROWID | string | |
| CHAR | string | |
| NCHAR | string | |
| CLOB | string | |
| NCLOB | string | |
| BLOB | bytes | |
| BFILE | bytes | BFILE is a data type used to store a locator (link) |
| | | to an external file, which is stored outside of the |
| | | database. Only the locator will be read from an |
| | | Oracle table and not the content of the external file. |
| Oracle Data Type | CDAP Schema Data Type | Comment |
| ------------------------------ | --------------------- | -----------------------------------------------------------|
| VARCHAR2 | string | |
| NVARCHAR2 | string | |
| VARCHAR | string | |
| NUMBER | string | For NUMBER types defined without a precision and scale. |
| | | Users can manually set output schema to map it to Decimal. |
| NUMBER | decimal | For NUMBER types defined with a precision and scale. |
| FLOAT | double | |
| LONG | string | |
| DATE | timestamp | |
| BINARY_FLOAT | float | |
| BINARY_DOUBLE | double | |
| TIMESTAMP | timestamp | |
| TIMESTAMP WITH TIME ZONE | string | |
| TIMESTAMP WITH LOCAL TIME ZONE | timestamp | |
| INTERVAL YEAR TO MONTH | string | |
| INTERVAL DAY TO SECOND | string | |
| RAW | bytes | |
| LONG RAW | bytes | |
| ROWID | string | |
| UROWID | string | |
| CHAR | string | |
| NCHAR | string | |
| CLOB | string | |
| NCLOB | string | |
| BLOB | bytes | |
| BFILE | bytes | BFILE is a data type used to store a locator (link) |
| | | to an external file, which is stored outside of the |
| | | database. Only the locator will be read from an |
| | | Oracle table and not the content of the external file. |


Example
Expand Down
16 changes: 16 additions & 0 deletions oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import io.cdap.cdap.api.annotation.MetadataProperty;
import io.cdap.cdap.api.annotation.Name;
import io.cdap.cdap.api.annotation.Plugin;
import io.cdap.cdap.api.data.schema.Schema;
import io.cdap.cdap.etl.api.FailureCollector;
import io.cdap.cdap.etl.api.batch.BatchSourceContext;
import io.cdap.cdap.etl.api.connector.Connector;
Expand Down Expand Up @@ -153,6 +154,21 @@ public void validate(FailureCollector collector) {
public String getTransactionIsolationLevel() {
return connection.getTransactionIsolationLevel();
}

@Override
protected void validateField(FailureCollector collector,
Schema.Field field, Schema actualFieldSchema, Schema expectedFieldSchema) {
// This change is needed to make sure that the pipeline upgrade continues to work post upgrade.
// Since the older handling of the precision less used to convert to the decimal type,
// and the new version would try to convert to the String type. In that case the output schema would
// contain Decimal(38, 0) (or something similar), and the code internally would try to identify
// the schema of the field(without precision and scale) as String.
if (Schema.LogicalType.DECIMAL.equals(expectedFieldSchema.getLogicalType())
&& actualFieldSchema.getType().equals(Schema.Type.STRING)) {
return;
}
super.validateField(collector, field, actualFieldSchema, expectedFieldSchema);
}
}

/**
Expand Down
12 changes: 10 additions & 2 deletions oracle-plugin/src/main/java/io/cdap/plugin/oracle/OracleSourceDBRecord.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,8 +223,16 @@ private void handleOracleSpecificType(ResultSet resultSet, StructuredRecord.Buil
recordBuilder.set(field.getName(), resultSet.getDouble(columnIndex));
} else {
if (precision == 0) {
// In case of precision less decimal convert the field to String type
recordBuilder.set(field.getName(), resultSet.getString(columnIndex));
Schema nonNullableSchema = field.getSchema().isNullable() ?
field.getSchema().getNonNullable() : field.getSchema();
if (Schema.LogicalType.DECIMAL.equals(nonNullableSchema.getLogicalType())) {
// Handle the field using the schema set in the output schema
BigDecimal decimal = resultSet.getBigDecimal(columnIndex, getScale(field.getSchema()));
recordBuilder.setDecimal(field.getName(), decimal);
} else {
// In case of Number defined without precision and scale convert to String type
recordBuilder.set(field.getName(), resultSet.getString(columnIndex));
}
} else {
// It's required to pass 'scale' parameter since in the case of Oracle, scale of 'BigDecimal' depends on the
// scale set in the logical schema. For example for value '77.12' if the scale set in the logical schema is
Expand Down
Loading