Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spatial filtering fails occasionally #28682

Closed
thomasneirynck opened this issue Feb 14, 2018 · 4 comments
Closed

Spatial filtering fails occasionally #28682

thomasneirynck opened this issue Feb 14, 2018 · 4 comments
Assignees
@nknize
Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes

Comments

@thomasneirynck
Copy link
Contributor

Using spatial filters (e.g. within, intersects) sometimes fails with the error message:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Shape with name [DE9BlWEBa-gv7TM2JoFM] found but missing geometry field"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "Shape with name [DE9BlWEBa-gv7TM2JoFM] found but missing geometry field"
  },
  "status": 500
}

This failure is not entirely consistent. It seems to fail about 30/50% of the time for small numbers of filters (<5), but about 100% of the time for large number of filters (>10).

I tested this on v6.1/6.2. It fails fairly predictably. This seems to work reliably on v5.x.

To reproduce:

  1. restore following snapshot into v6.1.3
    backup.zip

  2. run the following queries

a) with small number of filters. This fails about half the time.

GET geodata_uscities-small_3ccfef9cadc2-f747-da4f-abf9370d64e6/_search
 {
		"size": 0,
		"aggs": {
			"spatial_join": {
				"filters": {
					"filters": {
						"Dk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Dk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "intersects"
								}
							}
						},
						"DE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "DE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "intersects"
								}
							}
						}
					}
				}
			}
		}
	}

b) with large number of filters, this fails virtually all the time

{
		"size": 0,
		"aggs": {
			"spatial_join": {
				"filters": {
					"filters": {
						"Dk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Dk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"DE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "DE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"DU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "DU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"JE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "JE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"D09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "D09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"JU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "JU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Fk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Fk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"IE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "IE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"GU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "GU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Hk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Hk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"H09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "H09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"I09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "I09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Ik9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Ik9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"C09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "C09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"IU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "IU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Ek9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Ek9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Gk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Gk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"F09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "F09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"EE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "EE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"FE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "FE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"EU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "EU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"GE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "GE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"G09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "G09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"HU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "HU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"_09BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "_09BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"-E9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "-E9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"CE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "CE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"HE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "HE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"9E9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "9E9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"9U9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "9U9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"-U9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "-U9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"9k9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "9k9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Jk9BlWEBa-gv7TM2JoFN": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Jk9BlWEBa-gv7TM2JoFN",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"_U9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "_U9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"BU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "BU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"FU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "FU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"_E9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "_E9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"BE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "BE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"AE9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "AE9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Bk9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Bk9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"A09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "A09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"AU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "AU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"909BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "909BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"E09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "E09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"CU9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "CU9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Ck9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Ck9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"-k9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "-k9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"_k9BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "_k9BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"Ak9BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "Ak9BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"-09BlWEBa-gv7TM2JoBM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "-09BlWEBa-gv7TM2JoBM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						},
						"B09BlWEBa-gv7TM2JoFM": {
							"geo_shape": {
								"geometry": {
									"indexed_shape": {
										"index": "geodata_usastates_8f7fa23bfed4-a3b9-3599-0ac9a71dbf6d",
										"type": "feature",
										"id": "B09BlWEBa-gv7TM2JoFM",
										"path": "geometry"
									},
									"relation": "within"
								}
							}
						}
					}
				}
			}
		}
	}
@thomasneirynck thomasneirynck added the :Analytics/Geo Indexing, search aggregations of geo points and shapes label Feb 14, 2018
@thomasneirynck
Copy link
Contributor Author

cc @nknize

@DaveCTurner
Copy link
Contributor

Thanks for the detailed report @thomasneirynck. This was fixed by #28458, which will be released in 6.3.0.

@cmccowan
Copy link

cmccowan commented Feb 22, 2018 •
edited
Loading

Thanks @DaveCTurner , when do you expect to release the 6.3 deb version?

@DaveCTurner
Copy link
Contributor

I'm afraid we haven't committed to a definite date, but we're working towards this release now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nknize nknize

Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nknize @thomasneirynck @DaveCTurner @cmccowan