Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to parse timestamp values without milliseconds #694

Closed
brunborg opened this issue Feb 8, 2016 · 5 comments

Comments

@brunborg
Copy link

commented Feb 8, 2016

There is a bug in the parsing of timestamp values without milliseconds while using select

E.g. the value [2016-02-03T08:16:26.389] is successfully parsed, but the next value [2016-02-03T08:19] fails:

Failed with exception java.io.IOException:org.elasticsearch.hadoop.rest.EsHadoopParsingException: Cannot parse value [2016-02-03T08:19] for field [receivedTimestamp]

The error is consistent across different timestamp fields.

Using elasticsearch-hadoop-hive-2.2.0.jar

@costin

This comment has been minimized.

Copy link
Member

commented Feb 21, 2016

That's because there is no mapping defined in ES and thus, the first value is used to determine the time. Since the second value doesn't match exactly the first value, ES cannot properly map it and hence the exception.
To work-around this, one would define the mapping before hand as explained in the docs.

@costin costin added the question label Mar 22, 2016

@costin

This comment has been minimized.

Copy link
Member

commented Mar 22, 2016

As there hasn't been any update, closing the issue.

@costin costin closed this Mar 22, 2016

@srodiere

This comment has been minimized.

Copy link

commented Jun 24, 2016

I have same trouble and i don't understand why . There is something that i don't master at all about ES working

@wangxuegang1214

This comment has been minimized.

Copy link

commented Aug 3, 2017

'es.mapping.date.rich'='false'

@brianolsen87

This comment has been minimized.

Copy link

commented Oct 22, 2018

For a more concrete example.

#assuming you have a string field called customer and a date field called createdAt in elasticsearch

hive> DROP TABLE IF EXISTS my_table;

hive> CREATE EXTERNAL TABLE my_table (id string, customer string, createdat string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
    'es.resource' = 'my_table_es_index',
    'es.nodes' = 'es_url:9200',
    'es.read.metadata' = 'true',
    'es.mapping.names' = 'id:_metadata._id,customer:customer,createdat:createdAt',
    'es.mapping.date.rich' = 'false'
);

#print date as string
hive> select id, createdat, customer from my_table limit 10;

#convert the string described above to long (this type of logic can better reside inside a UDF)
hive> select id, unix_timestamp(createdat, "yyyy-MM-dd'T'HH:mm"), customer from my_table limit 10;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.