Support 64 bit ObjectIds

[jkerr5]

According to https://developers.arcgis.com/rest/services-reference/layer-feature-service-.htm

If the layer objectIdField does not have a length property or the length property is set to 4, the objectIdField is 32-bit. If the objectIdField has a length of 8, the objectIdField is 64-bit.

Can you add support for specifying the length of the ObjectId? Maybe by allowing the idFieldLength in the metadata?

[rgwozdz]

@jkerr5 thanks for posting this and providing the above link. I will put together a PR for 64 bit support in both FeatureServer and Winnow.

[rgwozdz]

Perhaps we could have the OBJECTID always be 64 bit?

[jkerr5]

That would be fine with me.

[jkerr5]

@rgwozdz any thoughts on when this might make it into a release?

[rgwozdz]

Hopefully relatively soon. I'm juggling it with a few other things. It's going to require a companion update/release of Winnow and then testing on all the ArcGIS clients, so it's a bit of work to coordinate - unless someone else wants the PR, I probably can't get to it until end of the week.

[jkerr5]

Thanks for the update @rgwozdz

[rgwozdz]

@jkerr5 - I'm realizing there is an impediment to implementation. JavaScript represents numbers using IEEE-754 double-precision (64 bit) format. It turns out that that give sus only 53 bits precision, which is around fifteen to sixteen decimal digits. When you get to numbers that large or larger, you end up getting approximations:

var bigNumber = 9023372036854775807
console.log(bigNumber); // 9023372036854776000

So while the link you provide indicates ArcGIS support of 64-bit OBJECTIDs, Koop can't readily deliver them due to limitations of JavaScript. A provider could supply a 64-bit integer as a string, but it would have to be converted to an integer prior to serving it to ArcGIS clients (as OBJECTIDs appear to still have to be integers for ArcGIS clients). That conversion could lead to a loss of precision, with the potential result being duplicate OBJECTIDs.

One way forward would be to have FeatureServer set the OBJECTID length to 8 (indicating 64-bit) but print a warning if a provider pushed through an OBJECTID that was greater that the unsigned 53-bit max value (9007199254740991).

How many digits where you hoping to support? Wondering if a max OBJECTID value of 9007199254740991 will suit your needs prior to undertaking the work.

[jkerr5]

The 53-bit max value of 9007199254740991 would be good. Anything more than the 32-bits we have now would be great :)

…

On Fri, Oct 12, 2018 at 7:47 PM Rich Gwozdz ***@***.***> wrote: @jkerr5 <https://github.com/jkerr5> - I'm realizing there is an impediment to implementation. JavaScript represents numbers using IEEE-754 double-precision (64 bit) format. It turns out that that give sus only 53 bits precision, which is around fifteen to sixteen decimal digits. When you get to numbers that large or larger, you end up getting approximations: var bigNumber = 9023372036854775807 console.log(bigNumber); // 9023372036854776000 So while the link you provide indicates ArcGIS support of 64-bit OBJECTIDs, Koop can't readily deliver them due to limitations of JavaScript. A provider could supply a 64-bit integer as a string, but it would have to be converted to an integer prior to serving it to ArcGIS clients (as OBJECTIDs appear to still have to be integers for ArcGIS clients). That conversion could lead to a loss of precision, with the potential result being duplicate OBJECTIDs. One way forward would be to have FeatureServer set the OBJECTID length to 8 (indicating 64-bit) but print a warning if a provider pushed through an OBJECTID that was greater that the unsigned 53-bit max value (9007199254740991). How many digits where you hoping to support? Wondering if a max OBJECTID value of 9007199254740991 will suit your needs prior to undertaking the work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP8Ais_yHOr_9otDds9PLHVkz8dCYPhks5ukNXLgaJpZM4Wu-S8> .

[jkerr5]

Hi @rgwozdz Any update on an ETA for this? We are working a project that needs bigger OBJECTIDs so were hoping this would make it in soon. Thanks.

[rgwozdz]

Hey @jkerr5 - sorry, I've been consumed with a few other big projects so I haven't been able to get to it. I may be freed up starting next week. Sorry, if you guys want to start your own PR, feel free. But it's still in my queue.

[rgwozdz]

@jkerr5 - finally got around to prototyping this, but I am finding that it doesn't work despite the documentation you noted in https://developers.arcgis.com/rest/services-reference/layer-feature-service-.htm.

I find that even when the length of the idField is set to 8, values over the 2147483647 max 32bit limit cause buggy behavior in ArcGIS Pro. Specifically, I'm finding that in the attribute table, the idField is assigned a value of 0 even though koop is sending a value of say 2147483650. Really large numbers appear to cause Pro to crash.

[jkerr5]

That's unfortunate to hear. I posted a question to the community https://community.esri.com/message/821822-can-arcgis-pro-support-64-bit-objectids

[rgwozdz]

Ideally, we would have an example feature service that implements the objectIdField length property and pumps out 64 bit integer IDs. Then we could analyze the responses, test in clients, and simulate the behavior in Koop.

[rgwozdz]

@jkerr5 - closing this given the response at
https://community.esri.com/message/821822-can-arcgis-pro-support-64-bit-objectids