Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reverse timestamp issue with cell timestamps #2193

Open
thileesf opened this issue Aug 2, 2019 · 1 comment
Open

Reverse timestamp issue with cell timestamps #2193

thileesf opened this issue Aug 2, 2019 · 1 comment

Comments

@thileesf
Copy link

@thileesf thileesf commented Aug 2, 2019

This is a request to document some limitations we came across with Bigtable, and a question about the use of bigtable-beam-import to copy data from HBase to Bigtable and vice-versa.

Documentation requests

In our HBase table we do Put operations with a reversed timestamp, i.e.,

Put p = new Put(row, column, qualifier, Long.MAX_VALUE - timeNowMillis, value);
table.put(p);

We do this to enforce a particular ordering, and this has worked fine with HBase.

When we used the bigtable-hbase-1.x client to do the same Put in Bigtable, the subsequent Get results all contained Long.MAX_VALUE timestamp. We traced it down to the com.google.cloud.bigtable.hbase.util.TimestampConverter which internally converts the time, and doesn't handle things well when the Put timestamp exceeds TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP. We are exploring ways to fix this.

I think it will be useful if you document this issue in CBT Docs, especially considering the HBase suggestions around this which others may be following.

Questions

Will you make the TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP public? Because one fix we are exploring is to use HBASE_EFFECTIVE_MAX_TIMESTAMP - timeNowMillis instead of Long.MAX_VALUE - timeNowMillis in our Puts; we are vary of computing it ourselves in case TimestampConverter.FACTOR changes.

Also, how does this work while using bigtable-beam-import:

a) Say, I have an HBase table with cells with timestamp in milliseconds. And I export this to sequencefiles using HBase's Export. When I import this sequencefile into Bigtable using bigtable-beam-import, will it do the hbase2bigtable() translation on the sequencefile cell timestamps?

b) If I export the data in Bigtable using bigtable-beam-import export, will it do the reverse translation using bigtable2hbase()? i.e., can I expect millisecond timestamps in the exported sequencefile or will it be microsecond timestamps?

c) How do these work if the HBase table had cells created with reverse timestamps, i.e., Long.MAX_VALUE - timeNowMillis?

@igorbernstein2

This comment has been minimized.

Copy link
Collaborator

@igorbernstein2 igorbernstein2 commented Aug 7, 2019

Hi,

You are absolutely right that we need to document this behavior. The issue stems from the fact that bigtable stores timestamps as microseconds while hbase uses milliseconds. The hbase adapter tries to adjust the discrepancy by multiplying the hbase value by 1000. However the range of values that we can store is narrowed by this conversion. The hbase adapter did not consider reverse timestamps and assumed that if the timestamp higher than the highest storable value then we can just use the highest possible value. When reading it would convert it to just Long.MAX_VALUE.

The sequence file importer/exporter will use the hbase client under the hood. So any value higher than HBASE_EFFECTIVE_MAX_TIMESTAMP will be read back as Long.MAX_VALUE.

Other than documenting the behavior, there is no easy solution to this. As it stands Long.MAX_VALUE - timeNowMillis pattern doesn't work with bigtable.

I'd like to understand your use case better this is the first time this has come up as an issue. Can you describe what the usecase for reverse timestamps in cell values?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.