Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping hive field to elastic geo_shape type #517

Closed
orlevi111 opened this issue Jul 22, 2015 · 6 comments

Comments

Projects
None yet
2 participants
@orlevi111
Copy link

commented Jul 22, 2015

Hi, I'm trying to read/write data from/to elasticsearch using Hive and I get ClassCastException when I try to read/write from/to index with geo_shape field.

I created this table in hive:

CREATE EXTERNAL TABLE features 
(
    SHAPE STRUCT<coordinates:array<array<array<double>>>,type:string>
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'hive_test/features',
              'es.index.auto.create' = 'false') ;

Elasticsearch index mapping:

PUT hive_test/features
{
    "properties":{
        "shape":{
            "type":"geo_shape"
        } 
    }
}
@costin

This comment has been minimized.

Copy link
Member

commented Jul 25, 2015

The exception occurs since the mapping for geo_shape (as with all non-primitive mappings) is a string. The reason behind that is that there's no dedicated type for it in Pig and thus it's up to the user to decide how to map it.
Change your mapping in Pig from nested array to chararray and rerun your task.

@orlevi111

This comment has been minimized.

Copy link
Author

commented Jul 26, 2015

I'm restricted to use hive.
Is there any way to use hive or do I have to use Pig in order to map to geo_shape field?

@costin

This comment has been minimized.

Copy link
Member

commented Jul 26, 2015

Neither Pig or Hive support a geo type; but that's not my point. Rather in your mapping define shape as a string and once it's loaded, then convert in Pig in whatever structure you want/need.

@orlevi111

This comment has been minimized.

Copy link
Author

commented Jul 27, 2015

I'm not sure I understood what you are suggesting.
But I need the mapping to be geo_shape becuase of geographic queries on the index.

@costin

This comment has been minimized.

Copy link
Member

commented Jul 27, 2015

Change your mapping in Pig from nested array to chararray and rerun your task.

CREATE EXTERNAL TABLE features (SHAPE chararray)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'hive_test/features',
              'es.index.auto.create' = 'false') ;
@costin

This comment has been minimized.

Copy link
Member

commented Aug 27, 2015

Closing the issue (no further follow-up).

@costin costin closed this Aug 27, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.