Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve foldMapMergeRedshiftSchemas signature #193

Merged
merged 1 commit into from
Sep 5, 2023

Conversation

oguzhanunlu
Copy link
Member

Details at #192

@oguzhanunlu oguzhanunlu self-assigned this Jul 10, 2023
@coveralls
Copy link

coveralls commented Jul 10, 2023

Pull Request Test Coverage Report for Build 5902784011

  • 15 of 15 (100.0%) changed or added relevant lines in 2 files are covered.
  • 10 unchanged lines in 6 files lost coverage.
  • Overall coverage decreased (-0.08%) to 80.943%

Files with Coverage Reduction New Missed Lines %
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/jsonschema/JsonPath.scala 1 85.71%
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/parquet/FieldValue.scala 1 81.36%
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/parquet/Migrations.scala 1 90.54%
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/bigquery/Row.scala 2 69.23%
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/jsonschema/circe/NumberCodecs.scala 2 70.37%
modules/core/src/main/scala/com.snowplowanalytics/iglu.schemaddl/jsonschema/circe/StringCodecs.scala 3 64.29%
Totals Coverage Status
Change from base Build 5474778699: -0.08%
Covered Lines: 1236
Relevant Lines: 1527

💛 - Coveralls

@@ -88,24 +88,24 @@ object ShredModelEntry {
if (s == NullCharacter) "\\\\N"
else s.replace('\t', ' ').replace('\n', ' ')

val extraCols = List(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you wrap the column name in quotes (e.g. """"schema_version""""). Then later in rdb-loader you remove the quotes again:

ShredModelEntry.extraCols.map(_._1.replaceAll(""""""", ""))

You might consider changing extraCols so that the column name is unquoted. This means RDB loader can be cleaner because you don't need to unquote. To compensate for this change, you could add the quoting somewhere inside showProps.


But there is more to consider!

By making this change, you have made extraCols part of the pulic-facing API of schema-ddl. So at the very least it should have scaladoc, and an explicit type annotation.

And then also ask your question.... is the type List[(String, String, String, String)] a sensible type for a public-facing value? Is there an alternative way of typing it, e.g. define a new case class?

Or alternatively, you could make extraCols private, and then define extraColNames as public:

val extraColNames: List[String] = extraCols.map(_._1)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks Ian, just pushed a new commit on both this PR and rdb loader PR

istreeter added a commit that referenced this pull request Aug 24, 2023
Copy link
Contributor

@istreeter istreeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider #194 , but apart from that I'd be happy to approve this.

oguzhanunlu pushed a commit that referenced this pull request Sep 4, 2023
@oguzhanunlu
Copy link
Member Author

hey @istreeter , I merged #194 into this branch and squashed all commits, does this look good to you? I'd like to proceed to the release and cut M1

@oguzhanunlu oguzhanunlu merged commit dd47b73 into develop Sep 5, 2023
2 checks passed
@oguzhanunlu oguzhanunlu deleted the improve-migration-signature branch September 5, 2023 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants