-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Needs utility for casting from qtemporals (or raw temporals) into datetimes #6
Comments
As a solution, I would propose to extend a It's possible to introspect the column types from qPython. Instances of >>> q.sync('table:: ([] pos:`d1`d2`d3;dates:(2001.01.01;2000.05.01;0Nd))')
>>> t = q.sync('table')
>>> print t.meta
metadata(dates=-14, qtype=98, pos=-11) You can check whether q type code correspond to a temporal value, by testing against the constants defined in the is_temporal = t.meta.dates in [QMONTH, QDATE, QDATETIME, QMINUTE, QSECOND, QTIME, QTIMESTAMP, QTIMESPAN] |
Thanks for the prompt reply. I stumbled onto Is there a particular reason for not, in the future, just defaulting all temporals into proper numpy datetimes? I know that numpy's storage of datetimes in inefficient compared to kdb's, but the current state of temporals is cumbersome to use. My bet is that either yall have a specific reason (which I'm curious to hear) or yall just haven't had the time. Please don't misinterpret this question as me being judgmental or something, I'm just genuinely curious. |
I've included an utility function in qtemporal for conversion between raw temporal array and numpy >>> raw = numpy.array([366, 121, qnull(QDATE)])
>>> print array_from_raw_qtemporal(raw, qtype = QDATE)
['2001-01-01' '2000-05-01' 'NaT'] Main reason for the representation in raw format was efficiency in memory representation of temporal vectors. Also we plan to allow user to configure mapping in final 1.0 release and choose between representation of temporal lists via either:
|
thanks for this one - will come in handy over here as well, maybe even the cheers, On 29 September 2014 11:30, Maciej Lach notifications@github.com wrote:
|
You might also want to checkout the SHA: b0ba797 commit. qPython can be instrumented to automatically represent q temporal vectors as numpy >>> v = q.sync("2001.01.01 2000.05.01 0Nd", numpy_temporals = True)
>>> print '%s dtype: %s qtype: %d: %s' % (type(v), v.dtype, v.meta.qtype, v)
<class 'qpython.qcollection.QList'> dtype: datetime64[D] qtype: -14: ['2001-01-01' '2000-05-01' 'NaT']
>>> v = q.sync("2000.01.04D05:36:57.600 0Np", numpy_temporals = True)
>>> print '%s dtype: %s qtype: %d: %s' % (type(v), v.dtype, v.meta.qtype, v)
<class 'qpython.qcollection.QList'> dtype: datetime64[ns] qtype: -12: ['2000-01-04T05:36:57.600000000+0100' 'NaT']
>>> v = q.sync("2001.01.01 2000.05.01 0Nd", numpy_temporals = False)
>>> print '%s dtype: %s qtype: %d: %s' % (type(v), v.dtype, v.meta.qtype, v)
<class 'qpython.qcollection.QTemporalList'> dtype: int32 qtype: -14: [2001-01-01 [metadata(qtype=-14)] 2000-05-01 [metadata(qtype=-14)]
NaT [metadata(qtype=-14)]]
>>> v = q.sync("2000.01.04D05:36:57.600 0Np", numpy_temporals = False)
>>> print '%s dtype: %s qtype: %d: %s' % (type(v), v.dtype, v.meta.qtype, v)
<class 'qpython.qcollection.QTemporalList'> dtype: int64 qtype: -12: [2000-01-04T05:36:57.600000000+0100 [metadata(qtype=-12)]
NaT [metadata(qtype=-12)]] For details please refer to documentation: |
Excellent, about to do a bit of timeseries analysis with qPython for the cheers, On 14 October 2014 08:33, Maciej Lach notifications@github.com wrote:
|
add publish workflow
Not sure if this is even possible given the IPC protocol used by kdb, but is there a way to transparently cast from q's temporals into numpy datetimes? Preferably, a flag in
QConnection.query
or a global config that can be set that would automatically cast a query's results which are temporal to numpy datetimes.I've made a flask-kdb extension using qpython, which for the record was a dream to work with (qpython and flask). To test out it's design, I made an in browser REPL link.
In doing so, the biggest issue I've had is the rendering of tables with datetimes in them:
I'd be welcome to some comments on what to do to resolve this. My current thinking is, if it's a kdb thing, the utility to convert the column could be as simple as running a
\t <whatever>
to get the metadata I need.Feel free to close this issue if this is a purely kdb thing that you can't fix.
The text was updated successfully, but these errors were encountered: