Question about Druid metadata

[noppanit]

We have a fairly large Druid data. So, to refresh the data takes a quiet a long time. I check Druid documentation and it seems like people recommend using merge and analysisTypes when querying metadata for faster result. I tried that and it seems to be the case.

My question is, currently caravel doesn't support those properties would it be better if Caravel supports that and what would be the impact of that? From what I've seen in the codebase https://github.com/airbnb/caravel/blob/master/caravel/models.py#L1560 it seems like Caravel only cares aboutcolumns.

Reference:
http://druid.io/docs/latest/querying/segmentmetadataquery.html

[mistercrunch]

Which version of Druid introduced segmentMetadata? Is it supported by pydruid?

We really want a good integration with Druid and will be happy reviewing/merging PRs that make improvements.

[noppanit]

segmentMetadata is supported by pydruid but it doesn't support merge right now.

Somebody has done a PR for pydruid already. I can ask them if it's possible to be merged.

druid-io/pydruid#64

If that's the case I can open a PR for improvement for Caravel.

Thanks.

[nishantmonu51]

druid-io/pydruid#64 has been fixed now.
@mistercrunch: I can look into this along with #2273, if it is not already done.

[mistercrunch]

@nishantmonu51 you should be able to assign issues to yourself and teammates now that you have more rights on the repo, also let me know your coworkers' Github handles and I'll be happy to give them "write" rights as well.

On my side I'll make sure to set up a watch on pydruid to keep up with the development there.

[mistercrunch]

Notice: this issue has been closed because it has been inactive for 420 days. Feel free to comment and request for this issue to be reopened.