Support and test mutation log queries at intermediate timestamps

[gdbelvin]

Standardize on quantums of time.Microsecond

[pavelkalinnikov]

Mostly nits, plus one "pleeeease".

[gdbelvin]

Ok, sooo...

I've removed all concepts of what "quantum" the storage layer has from tests, or anywhere else outside of the storage implementation itself. I think we've arrived at a place where the code in core can treat watermarks as opaque objects that it does NOT modify.

In order to have consistent behavior for the HighWatermarks test across storage implementations, I've settled on returning a timestamp "just beyond" the primary key of the highest item. Since we're using time.Time, it makes sense to use Add(1) regardless of the time fidelity of the storage layer.

It is now the storage layer's job to round timestamp inputs up to whatever quantum the storage layer itself decides to use. This preserves all the necessary semantics and (very helpfully) we now have tests that verify this behavior.

[pavelkalinnikov]

Is this TODO worth it? When a parameter is a list, it's already clear that it's a batch API.

[gdbelvin]

I think the idea was to create a distinct type to hold batches of mutations.
This would make the ReadLogs API much clearer since we could return []Batches rather than having the semi-awkard current situation of returning more than batchSize of mutations.

@mhutchinson -- your thoughts?

[mhutchinson]

If a batch is a first-class object then it seems subtle and indirect that it is constructed by passing in a bunch of EntryUpdates at the same time. This feels like it should be a slice at a minimum.

For the purposes of just this method I wouldn't go beyond turning the varargs mutation into a slice, but an API shouldn't be designed a method at a time so I'd need to re-review the whole API & future plans to make a call on the situation.

[pavelkalinnikov]

I'm surprized the SQL query takes time.Time as a parameter. My understanding was that it was a microseconds int.

Have you considered not using the DATETIME type? There is much details on how it works, which, together with details on how time.Time works, make me nervous about missing some unexpected corner cases like leap years/seconds or timezones, or keeping in mind this implicit "microsecond quantum" quirk.

Is MySQL DATETIME type one of the reasons you decided to use time.Time? I wonder if we could avoid this complexity as follows:

• Use BIGINT MySQL type for clarity
• Use time.Now().UnixNano() timestamp when writing
• Do not convert it back to time.Time when reading, just continue using the int
• Use Watermark type to wrap this int in a type-safe manner

[gdbelvin]

The big motivation for all these changes was managing the discrepancy between the mysql implementation (which used to be BIGINT with UnixNanos), and Spanner, which uses microseconds. These were two very different kinds of "ints", which caused their own problems.

• Trying to abstract away the time precision of the storage layer is how we got to time.Time.
• Migrating to time.Time has also produced a whole new suite of tests to ensure that storage implementations implement these APIs correctly, and have removed quite a few sneaky modifications of these watermarks in the code.
• In addition to abstracting away the time precision of the storage layer, I've also tried to match the precision of mysql with spanner (hence the use of DATETIME).

I think your concerns are very legitimate and worth discussing further.

[gdbelvin]

An interesting side effect of this PR is that is requires watermarks to have nanosecond precision.
Just thought I'd point that out for posterity.

[codecov]

Codecov Report

Merging #1380 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1380      +/-   ##
==========================================
+ Coverage   65.54%   65.61%   +0.06%     
==========================================
  Files          52       52              
  Lines        3959     3961       +2     
==========================================
+ Hits         2595     2599       +4     
+ Misses        969      964       -5     
- Partials      395      398       +3

Impacted Files	Coverage Δ
impl/sql/mutationstorage/mutation_logs.go	72.6% <100%> (+0.38%)	⬆️
core/sequencer/trillian_client.go	58.57% <0%> (-2.86%)	⬇️
core/sequencer/server.go	69.83% <0%> (-1.7%)	⬇️
impl/memory/mutation_logs.go	79.59% <0%> (+18.36%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36d41a9...b0363e7. Read the comment docs.

[mhutchinson]

@gdbelvin is this ready for review now, or are you planning on making the other changes regarding testing semantics for high watermark?

[gdbelvin]

TestHighWatermark has been reworked to avoid any dependency on the time fidelity of the storage implementation.

This also has the happy side effect of not imposing a particular time fidelity on the batch definition table. (Previous versions of this PR were going to force nanosecond fidelity)

[mhutchinson]

I feel like I saw Pavel use an empty string instead of the fmt.Sprintf description and the framework automatically generates a description?

[gdbelvin]

Hmm, I suppose that's true. For tests with the same name, the framework will postpend _$i to the name. That feels a bit hacky. Here I preserve an explicit link between the test name and the row it's testing.

[mhutchinson]

If a batch is a first-class object then it seems subtle and indirect that it is constructed by passing in a bunch of EntryUpdates at the same time. This feels like it should be a slice at a minimum.

For the purposes of just this method I wouldn't go beyond turning the varargs mutation into a slice, but an API shouldn't be designed a method at a time so I'd need to re-review the whole API & future plans to make a call on the situation.