'Longs' are converted to scientific notation when replicated to a shadow bucket

[cavalier123]

When an attribute with a large ‘long’ value (a little more precisely: an integer value of size greater than {say} a Java int primitive, i.e. > 2^31-1) is specified in a document, then Couchbase stores this field without using scientific notation in the application bucket. We have ascertained this by examining the operating system files used by Couchbase. However, when a document containing a ‘long’ is copied from the application bucket to a corresponding shadow bucket, then the ‘long’ will be stored using scientific notation. Again, we have ascertained this by examining the files used by Couchbase.

As an example of the problem: if I were to store the current Unix epoch time (1425295827909) in an attribute called ‘t_now’ in a Couchbase document, then the attribute be stored as:

“t_now” : 1425295827909

in the application bucket but as:

“t_now” : 1.425295827909e12

In the shadow bucket.

Note that this problem is not apparent when viewing documents in the Couchbase server admin console, as conversion from scientific notation to ‘plain’ (non-scientific) format appears to take place here.

This may not appear to be a big deal but is causing some subtle problems for us. The biggest problem we have occurs as follows:

(1) We write a document containing a ‘long’ to an application bucket;

(2) This 'long' is written as a ‘long’ in scientific notation in the document in the shadow bucket;

(3) This document is then replicated to a mobile device (running the Android version of Couchbase, Couchbase Lite). The ‘long’ is stored in scientific notation here;

(4) At some point, this document is replicated back from the mobile device to the application bucket. The ‘long’ that was previously stored in the document in ‘plain’ notation is now stored in scientific notation;

(5) We then have a process that reads documents from the application bucket using the Couchbase Java SDK. This process uses Jackson to deserialize ‘long’ attributes in documents to Java dates. This works with no problem with ‘long’ attributes that do not use scientific notation. However, the process fails when ‘long’s in scientific notation (i.e. those produced in {4}) are read, as Jackson interprets the ‘long’s in scientific notation as floating point numbers, which it will not deserialise to dates.

We think we can work around this in our application code but, as a general principle, it seems to me that attribute values stored in a document in a shadow bucket should be exactly the same (in terms of value and representation) as the corresponding values stored in the document in the application bucket.

We are using the Enterprise Editions of Couchbase Server, Gateway and Couchbase Lite (Android) at versions 3.0.2,, 1.0.3. and 1.0.3 (respectively), downloaded directly from the Couchbase web site.

[codersbet]

Can someone give the timeline of this. its causing a big issue on our end too

[adamcfraser]

I don't expect this to be fixed until at least 1.3.

[lynxnot]

This should be considered as a priority 1 bug!

[sethrosetter]

@adamcfraser I am unassigning myself. If you need a repro or help, please file a ticket in mobile-testkit.

[MatFl]

Will this ever be fixed? This is kind of a big issue!

[tleyden]

@MatFl this might be related to https://groups.google.com/forum/#!topic/golang-nuts/9SaVxumSc-Y, which could get fixed in the recent update to Go 1.9 for SG 1.5. Can you try to build Sync Gateway from the master branch using Go 1.9 and retest it?

The other workaround might be to get off of bucket shadowing and to switch to mobile convergence and use a single bucket. This is a brand new feature in the recently released SG 1.5.0 version.

I'll bring this issue up in our next dev team sprint planning meeting

[tleyden]

Closing as fixed by Go 1.9. @MatFl @cavalier123 Please re-open if you still run into issues