Schemaless dimension ingestion doesn't work when using spatial dimensions

[dclim]

If the list of dimensions is not specified (schemaless) but a spatial dimension is specified, the spatial dimension will be the only dimension ingested.

These are the relevant sections of code; spatial dimensions are now being added to the list of dimensions in DimensionSpec which prevents the introspection from happening in MapInputRowParser:

https://github.com/druid-io/druid-api/blob/master/src/main/java/io/druid/data/input/impl/DimensionsSpec.java#L93 and
https://github.com/druid-io/druid-api/blob/master/src/main/java/io/druid/data/input/impl/MapInputRowParser.java#L48

Is this something that we can/want to support, or is defining a spatial dimension now considered providing a list of dimensions?

[gianm]

It makes sense to me that spatial dimensions are dimensions, and if you include them, you don't get the dimensionless behavior.

That being said I think we will need some mode like "I want this core set of dimensions but I want everything else to be ingested too". So like a partially specified schema. This is useful for a bunch of cases:

• Specify spatial dimensions but have the rest be automatic
• Specify some fields as long dimensions but have other fields become string dimensions automatically
• Specify some dimensions so we can set a particular dimension sort order, but leave the rest out

I think this could all be satisfied with some flag like "discoverDimensions" that, if true, would enable dimension discovery even if the user has provided some dimensions and dimension exclusions already. For legacy reasons it should default to "true" if dimensions is null/empty and "false" otherwise. But if explicitly provided it should be respected.

[noppanit]

We specify spatial dimensions and have spatialDimensions and both are ingested correctly for 0.9.0. I guess this is fixed in 0.9.1.1 then? I would love to the flag to say discoverDimensions

[akashsagar]

I think the suggestion of discoverDimensions would be very useful. We too hit this issue. We have spatial data that can't be discovered and so we have to specify a spatial dimension but would like the rest to still be discovered. Schema discovery and evolution is a very attractive feature of Druid.

[uditsrn28]

Yeah we definitely need discoverDimensions feature. I am stuck due to this. Have some predefined fields and some un defined ones. Cant ingest all the dimensions using dimension less schema. useFieldDiscovery is one of the powerful feature of druid.

[dhuadaar]

Is there any development in pipeline on this issue or workaround. Even I am stuck due to this issue. Mentioning a spatialDimension is dropping all other dimensions I was expecting it to gather.

[dclim]

Thanks for the feedback all, AFAIK this hasn't been implemented yet. Will mark for next release to see if it gets more attention. Anyone here is free to contribute this patch!

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.