Fix SMB producer compatibility issues #4667
Conversation
| // version 1 is backwards compatible with version 0 | ||
| && (this.version <= 1 && other.version <= 1) |
There was a problem hiding this comment.
Shouldn't this be rolled back to original 0.11 code instead of being deleted ?
&& this.version == other.versionThere was a problem hiding this comment.
My thought process was that if a producer has already migrated to 0.12.1 and produced a few days of data, then upgrades to 0.12.2, they’ll have a mix of version: 1 and version: 0 in their metadata. If we make the version check strict again, the consumers will upgrade to 0.12.2 and break on version: 1 data :(
maybe i can keep the check the way it was (0 and 1 are compatible) and just make a note that the next version bump should be to 2.
There was a problem hiding this comment.
yes we should re-introduce some checks otherwise the release will accept any version
There was a problem hiding this comment.
Sounds good - updated
Codecov Report
@@ Coverage Diff @@
## main #4667 +/- ##
=======================================
Coverage 60.64% 60.65%
=======================================
Files 286 286
Lines 10453 10453
Branches 694 694
=======================================
+ Hits 6339 6340 +1
+ Misses 4114 4113 -1
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
| * bumping version: Scio versions prior to 0.12.1 perform a version compatibility check on SMB | ||
| * partitions which may fail if the SMB producer bumps to an incompatible version. | ||
| * | ||
| * <p>The next version bump should be to: 2 |
There was a problem hiding this comment.
the <p> tag was added by javafmt 🤷♀️
Unfortunately the concept of "version" in scio-smb was not well designed to start with so we find ourselves running into issues where consumers on older versions will break on producer updates. This is not an ideal solution but we can go forward with the mandate that all changes to the SMB metadata format must be backwards compatible... at least pending a thorough reworking of versioning semantics.