Dates reduced by a few nanoseconds when sent to KDB

[wwarby]

Somehow my dates are being corrupted by tiny margins when they are sent to a KDB process. In the very simplest form I can demonstrate the problem with this code:

conn.k('show', nodeq.timestamp(new Date(2019, 4, 10, 12)))

Output in KDB:

2019.05.10D10:59:59.999999872

The date is also off by an hour but I haven't tried dealing with the timezone issue yet - for now I need to deal with the weirdness in the nanoseconds. Any ideas why this is happening?

[wwarby]

No idea why this is happening but in the meantime I've worked around it with the following very ugly hack:

myFunc{[data]
    data:update timeColumn:`timestamp$`datetime$timeColumn from data;
    ...
    }

I can afford to do this because I don't need time precision less than 1 second in this particular scenario. Still, would like to know why this is happening so I don't have to use such an ugly hack

[michaelwittig]

Hi @wwarby
can you provide some additional information on the node, node-q and q version you use. As well as your operating system.

I see the output as expected:

2019.05.10D00:00:00.000000000

node-q: latest
node: 8.x
q: KDB+ 3.3 2016.03.14 Copyright (C) 1993-2016 Kx Systems
macOS 10.14.4 (18E226)

As you can see, I have stopped using q / kdb+ since quiet a few years and never updated to an up-to-date version :)

Regarding the time zone issue: set the timezone of your node process to UTC. q does not know about time zones. On Linux/macOS: TZ=UTC node ...

[wwarby]

Hi @michaelwittig,

Very good of you to keep answering questions on this repo considering you're no longer using q / kdb+ :)

So this does seem to be specific to some dates and not others - as far as I can tell, times that are precisely on the hour.

var nodeq = require('node-q');
nodeq.connect({host: 'localhost', port: 20010}, function(err, con) {
  let timestamp = new Date('2019-04-30T23:00:00.000Z');
  con.k('{string x}', nodeq.timestamp(timestamp), (err, result) => {
    console.log(result); // 2019.04.30D22:59:59.999999872
  });
  con.k('{string `timestamp$`datetime$x}', nodeq.timestamp(timestamp), (err, result) => {
    console.log(result); // Corrects the problem, but really hacky
  });
});

So the above should make it reproducible hopefully. I'm using Node v11.11, node-q latest, KDB+ latest (3.6), running Node on Windows 10.

[wwarby]

Interestingly I've also noticed that it's always off by precisely 128 nanoseconds - no more, no less.

[michaelwittig]

Could you, in lib/c.js line 707, replace

wlong(Long.fromNumber(86400000000000 * (x.getTime() / 86400000 - 10957)));

with

wlong(Long.fromNumber(Long.fromString("1000000", false, 10).multiply(Long.fromNumber(x.getTime())).subtract(Long.fromString("946684800000000000", false, 10))));

[wwarby]

Hi Michael, sorry it took me a couple of days to get back to you on this but yes, I can confirm that the patch your provided above solves the problem - thanks :)

So It's a bug in c.js then? I'll notify Kx but presumably you can fix it in the node-q library as c.js is an embedded library and not an npm dependency?

[michaelwittig]

c.js is heavily modified. I will provide a new release shortly

[michaelwittig]

released as node-q@2.4.4

Thanks for your bug report!

[wwarby]

Brilliant, thanks Michael