bigtable: emulator rejects puts with reversed timestamps

[nputnam]

#1790 Just a copy of this issue. It remains.

Client

bigtable-hbase-1.x-shaded version 1.6.0
hbase-shaded-client 1.4.5
Environment

OSX and Linux

Code

// Ideally, we'd just use Long.MAX_VALUE for this. However, the bigtable client needs to translate the value into
// microseconds as that is what Bigtable uses. This means they take the value we give them and they multiply it by
// 1000 which ends up blowing the size of a Long if we don't accommodate the expansion. Therefore, we set our max
// to the max value we can use but still accommodate the multiply by 1000
public static final long REVERSE_EPOCH = Long.MAX_VALUE / 1000L;

private long reverseTimestamp(long modTs) {
Preconditions.checkArgument(modTs >= 0);
return REVERSE_EPOCH - modTs;
}

private void doReversedTimestampPut(Table table) throws IOException {
byte[] key = "key".getBytes();

long timestamp = System.currentTimeMillis();
Put put = new Put(key, System.currentTimeMillis());
put.add(FAMILY, CREATION_TIMESTAMP_COL, reverseTimestamp(timestamp), Bytes.toBytes(timestamp));

table.put(put);

}
org.apache.hadoop.hbase.DoNotRetryIOException: Failed to perform operation. Operation='put', projectId='who_cares', tableName='not_me', rowKey='key'

at com.google.cloud.bigtable.hbase.AbstractBigtableTable.logAndCreateIOException(AbstractBigtableTable.java:536)
at com.google.cloud.bigtable.hbase.AbstractBigtableTable.put(AbstractBigtableTable.java:354)

Caused by: com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: UNKNOWN: invalid timestamp 9221789815520484000
at com.google.bigtable.repackaged.io.grpc.Status.asRuntimeException(Status.java:517)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.getBlockingResult(AbstractRetryingOperation.java:476)
at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.mutateRow(BigtableDataGrpcClient.java:236)
at com.google.cloud.bigtable.hbase.AbstractBigtableTable.put(AbstractBigtableTable.java:351)
... 28 more
Expected behavior

Reversed timestamps are allowed on mutations on the bigtable emulator.

Actual behavior

Timestamps after the year 10,000 are not allowed, which was introduced as a part of this change.

Additional context

We're doing timestamp reversing as described here:

If you usually retrieve the most recent records first, you can use a reversed timestamp in the row key by subtracting the timestamp from your programming language's maximum value for long integers (in Java, java.lang.Long.MAX_VALUE). With a reversed timestamp, the records will be ordered from most recent to least recent.

[igorbernstein2]

This is a bit surprising, all timestamp validation happens here:

google-cloud-go/bigtable/bttest/inmem.go

Lines 1190 to 1196 in e188592

	func (t *table) validTimestamp(ts int64) bool {
	if ts < minValidMilliSeconds || ts > maxValidMilliSeconds {
	return false
	}
	// Assume millisecond granularity is required.
	return ts%1000 == 0

The constants are defined here:

google-cloud-go/bigtable/bttest/inmem.go

Lines 62 to 67 in e188592

	minValidMilliSeconds = 0
	// MilliSeconds field of the max valid Timestamp.
	// Must match the max value of type TimestampMicros (int64)
	// truncated to the millis granularity by subtracting a remainder of 1000.
	maxValidMilliSeconds = math.MaxInt64 - math.MaxInt64%1000

math.MaxInt64 - math.MaxInt64%1000 evaluates to 9,223,372,036,854,775,000
The error message says that 9,221,789,815,520,484,000 is invalid, which is within the bounds

I wonder if you are using an older version of the emulator? Can you clarify how you've installed the emulator?

[nputnam]

Emulator is installed with the gcloud tools. Version says 2019.08.30.

[nputnam]

Any ideas here?

[jimfulton]

I verified that this is still broken with the version installed with gcloud. I verified that it is fixed in the version in git.

(My test program https://gist.github.com/jimfulton/69cca1687cc7bb043c28382c79c786db, I am not a Java developer and haven't used the Java HBase API before. :) )

So we need to see about getting gcloud to use a newer version.

[telpirion]

The emulator has been updated with the GitHub version since the last comment.

Please reopen this issue if the problem persists.