Skip to content

Fix table name returned for CASTed columns#352

Merged
labianchin merged 3 commits intospotify:masterfrom
rulle-io:ra/fix_cast_table_name
Aug 2, 2021
Merged

Fix table name returned for CASTed columns#352
labianchin merged 3 commits intospotify:masterfrom
rulle-io:ra/fix_cast_table_name

Conversation

@rulle-io
Copy link
Contributor

Problem

As reported by an internal user, when --sqlFile parameter is used and one of the columns is CASTed

SELECT
  CAST(date_test AS varchar) AS date_str,

dbeam throws an exception

[main] ERROR com.spotify.dbeam.jobs.ExceptionHandling 
- Failure: org.apache.avro.SchemaParseException: Empty name
at org.apache.avro.Schema.validateName(Schema.java:1525)
at org.apache.avro.Schema.access$400(Schema.java:87)
at org.apache.avro.Schema$Name.<init>(Schema.java:675)
at org.apache.avro.Schema.createRecord(Schema.java:212)
at org.apache.avro.SchemaBuilder$RecordBuilder.fields(SchemaBuilder.java:1805)
at com.spotify.dbeam.avro.JdbcAvroSchema.createAvroSchema(JdbcAvroSchema.java:113)
at com.spotify.dbeam.avro.JdbcAvroSchema.createSchemaByReadingOneRow(JdbcAvroSchema.java:86)
at com.spotify.dbeam.avro.BeamJdbcAvroSchema.generateAvroSchema(BeamJdbcAvroSchema.java:90)
at com.spotify.dbeam.avro.BeamJdbcAvroSchema.getAvroSchema(BeamJdbcAvroSchema.java:77)
at com.spotify.dbeam.avro.BeamJdbcAvroSchema.createSchema(BeamJdbcAvroSchema.java:54)
at com.spotify.dbeam.jobs.JdbcAvroJob.prepareExport(JdbcAvroJob.java:101)
at com.spotify.dbeam.jobs.JdbcAvroJob.runExport(JdbcAvroJob.java:145)
at com.spotify.dbeam.jobs.JdbcAvroJob.main(JdbcAvroJob.java:153)

It turned out that ResultSetMetaData.getTableName(...) returns an empty string for CASTed columns (on Postgres), which is not accepted later by org.apache.avro.Schema.

Changes

The change is to either return non-empty value or default one.

Tests:

  • "Unit tests are included"
  • "I successfully ran my test jobs with this code"

Checklist for PR author(s)

  • Changes are covered by unit tests (no major decrease in code coverage %) and/or integration tests.
  • Ensure code formating (use mvn com.coveo:fmt-maven-plugin:format org.codehaus.mojo:license-maven-plugin:update-file-header)
  • Document any relevant additions/changes in the appropriate spot in javadocs/docs/README.

@codecov
Copy link

codecov bot commented Jul 28, 2021

Codecov Report

Merging #352 (e070c9f) into master (7612a77) will increase coverage by 0.25%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #352      +/-   ##
============================================
+ Coverage     90.08%   90.34%   +0.25%     
- Complexity      230      233       +3     
============================================
  Files            25       25              
  Lines           908      911       +3     
  Branches         65       67       +2     
============================================
+ Hits            818      823       +5     
+ Misses           59       58       -1     
+ Partials         31       30       -1     

@rulle-io
Copy link
Contributor Author

@labianchin

@labianchin labianchin merged commit 4a0b56a into spotify:master Aug 2, 2021
@rulle-io rulle-io deleted the ra/fix_cast_table_name branch August 31, 2021 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants