Updated V7 generator to Draft04. #112

bgadrian · 2023-01-03T14:17:08Z

Updated V7 generator to enforce the monotonic property for ids generated in the same timestamp.
Updated tests and go docs.

generator.go

dylan-bourque · 2023-01-03T17:59:26Z

generator_test.go

+func makeTestNewV7TestVector() func(t *testing.T) {
+	return func(t *testing.T) {
+		pRand := make([]byte, 10)
+		//TODO make the comparison work with


Need to reconcile this TODO before thinking about merging this.

I will remove the TODO but I failed to do the actual validation on the random data compared with the example from the draft. At least for now I think a partial validation is better than nothing (the test now only asserts the first 15bytes).

codecov-commenter · 2023-01-03T18:03:29Z

Codecov Report

Base: 100.00% // Head: 100.00% // No change to project coverage 👍

Coverage data is based on head (6088057) compared to base (7b40032).
Patch coverage: 100.00% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #112   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            4         4           
  Lines          473       498   +25     
=========================================
+ Hits           473       498   +25

Impacted Files	Coverage Δ
generator.go	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

cameracker · 2023-01-06T18:12:54Z

Thanks for the submission @bgadrian! Would you mind rebasing this branch with master?

Also, thoughts on the code coverage loss?

convto · 2023-01-10T05:10:07Z

generator.go

 	u[1] = byte(ms >> 32)
 	u[2] = byte(ms >> 24)
 	u[3] = byte(ms >> 16)
 	u[4] = byte(ms >> 8)
 	u[5] = byte(ms)

+	//The 6th byte contains the version and partially rand_a data.
+	//We will lose the most significant bites from the clockSeq (with SetVersion), but it is ok, we need the least significant that contains the counter to ensure the monotonic property
+	binary.BigEndian.PutUint16(u[6:8], clockSeq) // set rand_a with clock seq which is random and monotonic


It may be better to make the API user-selectable whether to consider batch generation or not.
Because getClockSequence performs a mutex lock, and using it will result in worse performance and reduced generation capability.
For non-batch generation use cases, it is probably undesirable to have getClockSequence run, so a user-selectable API might be better.

(For example, the implementation related to draft allows breaking changes, so add isBatch to the NewV7() argument.)

So we moved this line from the top so that we can batch generate the UUID better, yes?

Can we call out in a comment here that this is done here specifically to support batching? I can see someone moving it around and unintentionally breaking that behavior.

@cameracker I moved that line for an improved readability. It was confusing to me first to fill bytes 8+ first, and then fill the 1-8 bytes. By moving that line specifically after or before the first bytes it would not affect the result, but all the lines after this one needs to be in order because of the overrides.

bgadrian · 2023-01-10T06:04:14Z

@convto If the user does not generate batches then it means it calls the method once each ms, so contention is not a problem. And without the monotonic seq the UUID is not v7 according to the specification. As alternative we could refactor or add a new method that uses atomic package instead of a mutex to improve the concurrency If needed.

On Tue, Jan 10, 2023, at 07:10, YuyaOkumura wrote: ***@***.**** commented on this pull request. In generator.go <#112 (comment)>: > u[1] = byte(ms >> 32) u[2] = byte(ms >> 24) u[3] = byte(ms >> 16) u[4] = byte(ms >> 8) u[5] = byte(ms) + //The 6th byte contains the version and partially rand_a data. + //We will lose the most significant bites from the clockSeq (with SetVersion), but it is ok, we need the least significant that contains the counter to ensure the monotonic property + binary.BigEndian.PutUint16(u[6:8], clockSeq) // set rand_a with clock seq which is random and monotonic It may be better to make the API user-selectable whether to consider batch generation or not. This is because `getClockSequence` performs a mutex lock, and using it will result in worse performance and reduced generation capability. For non-batch generation use cases, it is probably undesirable to have getClockSequence run, so a user-selectable API might be better. (For example, if breaking changes are allowed, adding `isBatch` to the NewV7 argument.) — Reply to this email directly, view it on GitHub <#112 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGKUMMCHZL5SX2JPLMRRSTWRTVLTANCNFSM6AAAAAATPXQRRM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

/*******************************/ ⚛ Bledea-Georgescu Adrian 🖧 Sofware Engineer ✍ https://coder.today 📧 ***@***.*** 🖥 github.com/bgadrian 💁 linkedin.com/in/bgadrian <https://www.linkedin.com/in/bgadrian/>

convto · 2023-01-10T07:09:02Z

@bgadrian
The Rev04 draft monotonic counter specification was defined with SHOULD and MAY requirement levels, so I wanted to give the user a flexible option. But as you commented, it doesn't seem to be much of a problem.

Thanks for your reply!

bgadrian · 2023-01-11T07:29:24Z

Thanks for the submission @bgadrian! Would you mind rebasing this branch with master?

Also, thoughts on the code coverage loss?

Hello, I have addressed the comments, restored the code coverage and rebased with the master.

cameracker · 2023-01-23T18:39:56Z

Hi @bgadrian ! I'm still planning on reviewing and accepting this contribution but haven't had the time to study the new updates to the draft to check for correct implementation. I'll do my best to get to it this week, I appreciate your patience.

cameracker · 2023-01-23T18:48:37Z

Also, I am tentatively planning on putting out a release as soon as this is merged.

Another thing @bgadrian , a couple of PRs have been merged to Master. One meaningful PR is the change to how coverage is collected. The addition of Generator options may have a small impact on this PR but unclear. Would you rather me update this PR for you or would you like to process these updates yourself?

generator.go

cameracker · 2023-01-24T00:30:26Z

generator.go

+	   |                            rand_b                             |
+	   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */
+
+	ms, clockSeq, err := g.getClockSequence(true)


Ok, and then just to make sure I understand: this isnt really strictly needed for the PR, it looks like this is just a refactor to move this calculation into the clock sequence rather than just doing it here to meet the ms requirement for this specific uuid. Is that the case? I don't have a strong preference here but I'll say that the boolean flag parameteter on getClockSequence is slightly more mysterious if we're trying to understand "why" that flag exists. It's private so it's not a big deal and I won't to ask for a reshuffle if other maintainers are ok with it.

Yes, I wanted to reuse the code sequencer and the mutex, but with a different timestamp, hence the flag.

generator.go

cameracker · 2023-01-24T00:34:52Z

generator.go

 	u[1] = byte(ms >> 32)
 	u[2] = byte(ms >> 24)
 	u[3] = byte(ms >> 16)
 	u[4] = byte(ms >> 8)
 	u[5] = byte(ms)

+	//The 6th byte contains the version and partially rand_a data.
+	//We will lose the most significant bites from the clockSeq (with SetVersion), but it is ok, we need the least significant that contains the counter to ensure the monotonic property
+	binary.BigEndian.PutUint16(u[6:8], clockSeq) // set rand_a with clock seq which is random and monotonic


So we moved this line from the top so that we can batch generate the UUID better, yes?

Can we call out in a comment here that this is done here specifically to support batching? I can see someone moving it around and unintentionally breaking that behavior.

cameracker · 2023-01-24T00:42:24Z

generator.go

@@ -272,28 +281,50 @@ func (g *Gen) getClockSequence() (uint64, uint16, error) {
 // NewV7 returns a k-sortable UUID based on the current millisecond precision
 // UNIX epoch and 74 bits of pseudorandom data.
 //
-// This is implemented based on revision 03 of the Peabody UUID draft, and may
+// This is implemented based on revision 04 of the Peabody UUID draft, and may
 // be subject to change pending further revisions. Until the final specification
 // revision is finished, changes required to implement updates to the spec will
 // not be considered a breaking change. They will happen as a minor version
 // releases until the spec is final.


Given that this draft focuses on being more tentative about how strongly the implementations need to respect monotonicity of the increments vs unguessability, do we owe it to users to be explicit about which behavior we're leaning towards in the implementation?

cameracker · 2023-01-24T00:47:35Z

Ok, I completed a review. Sorry it took me so long. And thank you so much for the contribution.

Last request: Could we update the README.md to reflect which version of the Draft we're implementing for v6 and v7?

As an overall comment, I believe this PR correctly implements the v6 and v7 UUIDs to specification, but I'm getting the sense that we're not being as clear as we could be on which "MAY" "SHOULD" behaviors we chose to address in this implementation and worry that some user is going to pick up those UUIDs and run into "undefined behavior" sort of problems. What do you think? Should we be more explicit on our approach anywhere? @convto @bgadrian @dylan-bourque

bgadrian · 2023-01-26T07:43:37Z

I have merged with the latest master, updated the Readme and addressed some comments.

As for being explicit or not, I think the v4 specifications is not a MAY or SHOULD, it is mandatory (SHOULD) to ensure the monotonic property

Additionally, care SHOULD be taken to ensure UUIDs generated in batches are also monotonic. That is, if one-thousand UUIDs are generated for the same timestamp; there is sufficient logic for organizing the creation order of those one-thousand UUIDs.

But the specs does not enforce which algorithm to use

MAY utilize a monotonic counter

The draft states that

For single-node UUID implementations that do not need to create batches of UUIDs,

This indeed makes the Batching optional, which is confusing, but the problem is that, the users will not know if they need or not batching most likely, I presume most real world scenarios of generating UUIDs are based on events that cannot be controlled (new users, new resources), so the "need" or "not need" of batching cannot be guaranteed, only presumed that is ok 99.99% of the time.

Updated V7 generator to Draft04.

8503ae3

dylan-bourque reviewed Jan 3, 2023

View reviewed changes

comment fixes

ec89839

convto reviewed Jan 10, 2023

View reviewed changes

bgadrian added 2 commits January 11, 2023 09:17

Merge branch 'master' into uuidv7-draft4-update

59adcc2

extend test coverage for failing new rand call

85bb292

cameracker requested changes Jan 24, 2023

View reviewed changes

bgadrian added 3 commits January 26, 2023 09:17

Merge branch 'master' into uuidv7-draft4-update

3707ceb

update readme

c45ce6a

fix more comments

6088057

cameracker approved these changes Jan 26, 2023

View reviewed changes

cameracker merged commit 8345c9a into gofrs:master Jan 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated V7 generator to Draft04. #112

Updated V7 generator to Draft04. #112

bgadrian commented Jan 3, 2023

dylan-bourque Jan 3, 2023

bgadrian Jan 4, 2023

codecov-commenter commented Jan 3, 2023 •

edited

Loading

cameracker commented Jan 6, 2023

convto Jan 10, 2023 •

edited

Loading

cameracker Jan 24, 2023

bgadrian Jan 26, 2023

bgadrian commented Jan 10, 2023 via email

convto commented Jan 10, 2023 •

edited

Loading

bgadrian commented Jan 11, 2023

cameracker commented Jan 23, 2023

cameracker commented Jan 23, 2023

cameracker Jan 24, 2023

bgadrian Jan 26, 2023

cameracker Jan 24, 2023

cameracker Jan 24, 2023

cameracker commented Jan 24, 2023 •

edited

Loading

bgadrian commented Jan 26, 2023

Updated V7 generator to Draft04. #112

Updated V7 generator to Draft04. #112

Conversation

bgadrian commented Jan 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jan 3, 2023 • edited Loading

Codecov Report

cameracker commented Jan 6, 2023

convto Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgadrian commented Jan 10, 2023 via email

convto commented Jan 10, 2023 • edited Loading

bgadrian commented Jan 11, 2023

cameracker commented Jan 23, 2023

cameracker commented Jan 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cameracker commented Jan 24, 2023 • edited Loading

bgadrian commented Jan 26, 2023

codecov-commenter commented Jan 3, 2023 •

edited

Loading

convto Jan 10, 2023 •

edited

Loading

convto commented Jan 10, 2023 •

edited

Loading

cameracker commented Jan 24, 2023 •

edited

Loading