Fix fields missing from the result hits (fixes #692)

[bartbroere]

This is a hotfix that solves the issues in our data warehouse. Some more work is needed to reproduce this issue with one of the test datasets. Also see the analysis in the related issue #692

To do:

• Regenerate notebook tests

Fixes #692

[pquentin]

buildkite test this please

[pquentin]

buildkite test this please

[pquentin]

buildkite test this please

[bartbroere]

Thanks for running this. I reformatted with black, and noticed that the ordering of the values in the Cities column is non-deterministic, so I'll need to address that as well.

[pquentin]

buildkite test this please

[pquentin]

Thanks. Would you mind pasting here a JSON response from Elasticsearch where _source is a list and fields is set? That would help reviewing this.

[bartbroere]

Thanks. Would you mind pasting here a JSON response from Elasticsearch where _source is a list and fields is set? That would help reviewing this.

Yes, no problem:

Request:

{"_source": ["AvgTicketPrice", "Cancelled", "Carrier", "Cities", "Dest", "DestAirportID", "DestCityName", "DestCountry", "DestLocation", "DestRegion", "DestWeather", "DistanceKilometers", "DistanceMiles", "FlightDelay", "FlightDelayMin", "FlightDelayType", "FlightNum", "FlightTimeHour", "FlightTimeMin", "Origin", "OriginAirportID", "OriginCityName", "OriginCountry", "OriginLocation", "OriginRegion", "OriginWeather", "dayOfWeek", "timestamp"], "sort": [{"_doc": "asc"}], "size": 5000, "track_total_hits": false, "pit": {"id": "uMyMBAEHZmxpZ2h0cxZYMTUxQ2txZVQ4aUs2NU13aWZ4ZVFRABZvZFFHLThJSlF6aVdDUklqRm94UE1RAAAAAAAAAABEFmxoa2haTkdGU2h1X01yb2xaamhDbmcAARZYMTUxQ2txZVQ4aUs2NU13aWZ4ZVFRAAA=", "keep_alive": "3m"}, "fields": ["AvgTicketPrice", "Cancelled", "Carrier", "Cities", "Dest", "DestAirportID", "DestCityName", "DestCountry", "DestLocation", "DestRegion", "DestWeather", "DistanceKilometers", "DistanceMiles", "FlightDelay", "FlightDelayMin", "FlightDelayType", "FlightNum", "FlightTimeHour", "FlightTimeMin", "Origin", "OriginAirportID", "OriginCityName", "OriginCountry", "OriginLocation", "OriginRegion", "OriginWeather", "dayOfWeek", "timestamp"]}

Response:

{"pit_id": "uMyMBAEHZmxpZ2h0cxZYMTUxQ2txZVQ4aUs2NU13aWZ4ZVFRABZvZFFHLThJSlF6aVdDUklqRm94UE1RAAAAAAAAAABEFmxoa2haTkdGU2h1X01yb2xaamhDbmcAARZYMTUxQ2txZVQ4aUs2NU13aWZ4ZVFRAAA=", "took": 147, "timed_out": false, "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0}, "hits": {"max_score": null, "hits": [{"_index": "flights", "_id": "0", "_score": null, "_source": {"FlightNum": "9HY9SWR", "DestCountry": "AU", "OriginWeather": "Sunny", "OriginCityName": "Frankfurt am Main", "AvgTicketPrice": 841.2656419677076, "DistanceMiles": 10247.856675613455, "FlightDelay": false, "DestWeather": "Rain", "Dest": "Sydney Kingsford Smith International Airport", "FlightDelayType": "No Delay", "OriginCountry": "DE", "dayOfWeek": 0, "DistanceKilometers": 16492.32665375846, "timestamp": "2018-01-01T00:00:00", "DestLocation": {"lat": "-33.94609833", "lon": "151.177002"}, "DestAirportID": "SYD", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 1030.7704158599038, "Origin": "Frankfurt am Main Airport", "OriginLocation": {"lat": "50.033333", "lon": "8.570556"}, "DestRegion": "SE-BD", "OriginAirportID": "FRA", "OriginRegion": "DE-HE", "DestCityName": "Sydney", "FlightTimeHour": 17.179506930998397, "FlightDelayMin": 0}, "fields": {"FlightNum": ["9HY9SWR"], "Origin": ["Frankfurt am Main Airport"], "OriginLocation": [{"coordinates": [8.570556, 50.033333], "type": "Point"}], "DestLocation": [{"coordinates": [151.177002, -33.94609833], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [10247.856], "FlightTimeMin": [1030.7704], "OriginWeather": ["Sunny"], "dayOfWeek": [0], "AvgTicketPrice": [841.2656], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["DE-HE"], "DestAirportID": ["SYD"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T00:00:00"], "Dest": ["Sydney Kingsford Smith International Airport"], "FlightTimeHour": [17.179506], "Cancelled": [false], "DistanceKilometers": [16492.326], "OriginCityName": ["Frankfurt am Main"], "DestWeather": ["Rain"], "OriginCountry": ["DE"], "Cities": ["Sydney", "Frankfurt am Main"], "DestCountry": ["AU"], "DestRegion": ["SE-BD"], "DestCityName": ["Sydney"], "OriginAirportID": ["FRA"]}, "sort": [0, 0]}, {"_index": "flights", "_id": "1", "_score": null, "_source": {"FlightNum": "X98CCZO", "DestCountry": "IT", "OriginWeather": "Clear", "OriginCityName": "Cape Town", "AvgTicketPrice": 882.9826615595518, "DistanceMiles": 5482.606664853586, "FlightDelay": false, "DestWeather": "Sunny", "Dest": "Venice Marco Polo Airport", "FlightDelayType": "No Delay", "OriginCountry": "ZA", "dayOfWeek": 0, "DistanceKilometers": 8823.40014044213, "timestamp": "2018-01-01T18:27:00", "DestLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestAirportID": "VE05", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 464.3894810759016, "Origin": "Cape Town International Airport", "OriginLocation": {"lat": "-33.96480179", "lon": "18.60169983"}, "DestRegion": "IT-34", "OriginAirportID": "CPT", "OriginRegion": "SE-BD", "DestCityName": "Venice", "FlightTimeHour": 7.73982468459836, "FlightDelayMin": 0}, "fields": {"FlightNum": ["X98CCZO"], "Origin": ["Cape Town International Airport"], "OriginLocation": [{"coordinates": [18.60169983, -33.96480179], "type": "Point"}], "DestLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5482.6064], "FlightTimeMin": [464.3895], "OriginWeather": ["Clear"], "dayOfWeek": [0], "AvgTicketPrice": [882.98267], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["VE05"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T18:27:00"], "Dest": ["Venice Marco Polo Airport"], "FlightTimeHour": [7.739825], "Cancelled": [false], "DistanceKilometers": [8823.4], "OriginCityName": ["Cape Town"], "DestWeather": ["Sunny"], "OriginCountry": ["ZA"], "Cities": ["Venice", "Cape Town"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Venice"], "OriginAirportID": ["CPT"]}, "sort": [1, 1]}, {"_index": "flights", "_id": "2", "_score": null, "_source": {"FlightNum": "UFK2WIZ", "DestCountry": "IT", "OriginWeather": "Rain", "OriginCityName": "Venice", "AvgTicketPrice": 190.6369038508356, "DistanceMiles": 0.0, "FlightDelay": false, "DestWeather": "Cloudy", "Dest": "Venice Marco Polo Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 0.0, "timestamp": "2018-01-01T17:11:14", "DestLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestAirportID": "VE05", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 0.0, "Origin": "Venice Marco Polo Airport", "OriginLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestRegion": "IT-34", "OriginAirportID": "VE05", "OriginRegion": "IT-34", "DestCityName": "Venice", "FlightTimeHour": 0.0, "FlightDelayMin": 0}, "fields": {"FlightNum": ["UFK2WIZ"], "Origin": ["Venice Marco Polo Airport"], "OriginLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "DestLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [0.0], "FlightTimeMin": [0.0], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [190.6369], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["IT-34"], "DestAirportID": ["VE05"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T17:11:14"], "Dest": ["Venice Marco Polo Airport"], "FlightTimeHour": [0.0], "Cancelled": [false], "DistanceKilometers": [0.0], "OriginCityName": ["Venice"], "DestWeather": ["Cloudy"], "OriginCountry": ["IT"], "Cities": ["Venice", "Venice"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Venice"], "OriginAirportID": ["VE05"]}, "sort": [2, 2]}, {"_index": "flights", "_id": "3", "_score": null, "_source": {"FlightNum": "EAYQW69", "DestCountry": "IT", "OriginWeather": "Thunder & Lightning", "OriginCityName": "Naples", "AvgTicketPrice": 181.69421554118, "DistanceMiles": 345.31943877289535, "FlightDelay": true, "DestWeather": "Clear", "Dest": "Treviso-Sant\'Angelo Airport", "FlightDelayType": "Weather Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 555.7377668725265, "timestamp": "2018-01-01T10:33:28", "DestLocation": {"lat": "45.648399", "lon": "12.1944"}, "DestAirportID": "TV01", "Carrier": "Kibana Airlines", "Cancelled": true, "FlightTimeMin": 222.74905899019436, "Origin": "Naples International Airport", "OriginLocation": {"lat": "40.886002", "lon": "14.2908"}, "DestRegion": "IT-34", "OriginAirportID": "NA01", "OriginRegion": "IT-72", "DestCityName": "Treviso", "FlightTimeHour": 3.712484316503239, "FlightDelayMin": 180}, "fields": {"FlightNum": ["EAYQW69"], "Origin": ["Naples International Airport"], "OriginLocation": [{"coordinates": [14.2908, 40.886002], "type": "Point"}], "DestLocation": [{"coordinates": [12.1944, 45.648399], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [345.31943], "FlightTimeMin": [222.74905], "OriginWeather": ["Thunder & Lightning"], "dayOfWeek": [0], "AvgTicketPrice": [181.69421], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [180], "OriginRegion": ["IT-72"], "DestAirportID": ["TV01"], "FlightDelayType": ["Weather Delay"], "timestamp": ["2018-01-01T10:33:28"], "Dest": ["Treviso-Sant\'Angelo Airport"], "FlightTimeHour": [3.7124844], "Cancelled": [true], "DistanceKilometers": [555.7378], "OriginCityName": ["Naples"], "DestWeather": ["Clear"], "OriginCountry": ["IT"], "Cities": ["Treviso", "Naples"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Treviso"], "OriginAirportID": ["NA01"]}, "sort": [3, 3]}, {"_index": "flights", "_id": "4", "_score": null, "_source": {"FlightNum": "58U013N", "DestCountry": "CN", "OriginWeather": "Damaging Wind", "OriginCityName": "Mexico City", "AvgTicketPrice": 730.041778346198, "DistanceMiles": 8300.428124665925, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Xi\'an Xianyang International Airport", "FlightDelayType": "No Delay", "OriginCountry": "MX", "dayOfWeek": 0, "DistanceKilometers": 13358.24419986236, "timestamp": "2018-01-01T05:13:00", "DestLocation": {"lat": "34.447102", "lon": "108.751999"}, "DestAirportID": "XIY", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 785.7790705801389, "Origin": "Licenciado Benito Juarez International Airport", "OriginLocation": {"lat": "19.4363", "lon": "-99.072098"}, "DestRegion": "SE-BD", "OriginAirportID": "AICM", "OriginRegion": "MX-DIF", "DestCityName": "Xi\'an", "FlightTimeHour": 13.096317843002314, "FlightDelayMin": 0}, "fields": {"FlightNum": ["58U013N"], "Origin": ["Licenciado Benito Juarez International Airport"], "OriginLocation": [{"coordinates": [-99.072098, 19.4363], "type": "Point"}], "DestLocation": [{"coordinates": [108.751999, 34.447102], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [8300.428], "FlightTimeMin": [785.77905], "OriginWeather": ["Damaging Wind"], "dayOfWeek": [0], "AvgTicketPrice": [730.04175], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["MX-DIF"], "DestAirportID": ["XIY"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T05:13:00"], "Dest": ["Xi\'an Xianyang International Airport"], "FlightTimeHour": [13.096318], "Cancelled": [false], "DistanceKilometers": [13358.244], "OriginCityName": ["Mexico City"], "DestWeather": ["Clear"], "OriginCountry": ["MX"], "Cities": ["Xi\'an", "Mexico City"], "DestCountry": ["CN"], "DestRegion": ["SE-BD"], "DestCityName": ["Xi\'an"], "OriginAirportID": ["AICM"]}, "sort": [4, 4]}, {"_index": "flights", "_id": "5", "_score": null, "_source": {"FlightNum": "XEJ78I2", "DestCountry": "IT", "OriginWeather": "Rain", "OriginCityName": "Edmonton", "AvgTicketPrice": 418.1520890531832, "DistanceMiles": 4891.315227492962, "FlightDelay": false, "DestWeather": "Thunder & Lightning", "Dest": "Genoa Cristoforo Colombo Airport", "FlightDelayType": "No Delay", "OriginCountry": "CA", "dayOfWeek": 0, "DistanceKilometers": 7871.808813474433, "timestamp": "2018-01-01T01:43:03", "DestLocation": {"lat": "44.4133", "lon": "8.8375"}, "DestAirportID": "GE01", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 393.5904406737217, "Origin": "Edmonton International Airport", "OriginLocation": {"lat": "53.30970001", "lon": "-113.5800018"}, "DestRegion": "IT-42", "OriginAirportID": "CYEG", "OriginRegion": "CA-AB", "DestCityName": "Genova", "FlightTimeHour": 6.559840677895361, "FlightDelayMin": 0}, "fields": {"FlightNum": ["XEJ78I2"], "Origin": ["Edmonton International Airport"], "OriginLocation": [{"coordinates": [-113.5800018, 53.30970001], "type": "Point"}], "DestLocation": [{"coordinates": [8.8375, 44.4133], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4891.3154], "FlightTimeMin": [393.59045], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [418.1521], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["CA-AB"], "DestAirportID": ["GE01"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T01:43:03"], "Dest": ["Genoa Cristoforo Colombo Airport"], "FlightTimeHour": [6.5598407], "Cancelled": [false], "DistanceKilometers": [7871.8086], "OriginCityName": ["Edmonton"], "DestWeather": ["Thunder & Lightning"], "OriginCountry": ["CA"], "Cities": ["Genova", "Edmonton"], "DestCountry": ["IT"], "DestRegion": ["IT-42"], "DestCityName": ["Genova"], "OriginAirportID": ["CYEG"]}, "sort": [5, 5]}, {"_index": "flights", "_id": "6", "_score": null, "_source": {"FlightNum": "EVARI8I", "DestCountry": "CH", "OriginWeather": "Clear", "OriginCityName": "Zurich", "AvgTicketPrice": 180.24681638061213, "DistanceMiles": 0.0, "FlightDelay": true, "DestWeather": "Hail", "Dest": "Zurich Airport", "FlightDelayType": "Security Delay", "OriginCountry": "CH", "dayOfWeek": 0, "DistanceKilometers": 0.0, "timestamp": "2018-01-01T13:49:53", "DestLocation": {"lat": "47.464699", "lon": "8.54917"}, "DestAirportID": "ZRH", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 300.0, "Origin": "Zurich Airport", "OriginLocation": {"lat": "47.464699", "lon": "8.54917"}, "DestRegion": "CH-ZH", "OriginAirportID": "ZRH", "OriginRegion": "CH-ZH", "DestCityName": "Zurich", "FlightTimeHour": 5.0, "FlightDelayMin": 300}, "fields": {"FlightNum": ["EVARI8I"], "Origin": ["Zurich Airport"], "OriginLocation": [{"coordinates": [8.54917, 47.464699], "type": "Point"}], "DestLocation": [{"coordinates": [8.54917, 47.464699], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [0.0], "FlightTimeMin": [300.0], "OriginWeather": ["Clear"], "dayOfWeek": [0], "AvgTicketPrice": [180.24681], "Carrier": ["JetBeats"], "FlightDelayMin": [300], "OriginRegion": ["CH-ZH"], "DestAirportID": ["ZRH"], "FlightDelayType": ["Security Delay"], "timestamp": ["2018-01-01T13:49:53"], "Dest": ["Zurich Airport"], "FlightTimeHour": [5.0], "Cancelled": [false], "DistanceKilometers": [0.0], "OriginCityName": ["Zurich"], "DestWeather": ["Hail"], "OriginCountry": ["CH"], "Cities": ["Zurich", "Zurich"], "DestCountry": ["CH"], "DestRegion": ["CH-ZH"], "DestCityName": ["Zurich"], "OriginAirportID": ["ZRH"]}, "sort": [6, 6]}, {"_index": "flights", "_id": "7", "_score": null, "_source": {"FlightNum": "1IRBW25", "DestCountry": "CA", "OriginWeather": "Thunder & Lightning", "OriginCityName": "Rome", "AvgTicketPrice": 585.1843103083941, "DistanceMiles": 4203.1829639346715, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Ottawa Macdonald-Cartier International Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 6764.367283910481, "timestamp": "2018-01-01T04:54:59", "DestLocation": {"lat": "45.32249832", "lon": "-75.66919708"}, "DestAirportID": "YOW", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 614.9424803554983, "Origin": "Ciampino___G. B. Pastine International Airport", "OriginLocation": {"lat": "41.7994", "lon": "12.5949"}, "DestRegion": "CA-ON", "OriginAirportID": "RM12", "OriginRegion": "IT-62", "DestCityName": "Ottawa", "FlightTimeHour": 10.249041339258305, "FlightDelayMin": 0}, "fields": {"FlightNum": ["1IRBW25"], "Origin": ["Ciampino___G. B. Pastine International Airport"], "OriginLocation": [{"coordinates": [12.5949, 41.7994], "type": "Point"}], "DestLocation": [{"coordinates": [-75.66919708, 45.32249832], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4203.183], "FlightTimeMin": [614.9425], "OriginWeather": ["Thunder & Lightning"], "dayOfWeek": [0], "AvgTicketPrice": [585.1843], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["IT-62"], "DestAirportID": ["YOW"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T04:54:59"], "Dest": ["Ottawa Macdonald-Cartier International Airport"], "FlightTimeHour": [10.249042], "Cancelled": [false], "DistanceKilometers": [6764.367], "OriginCityName": ["Rome"], "DestWeather": ["Clear"], "OriginCountry": ["IT"], "Cities": ["Ottawa", "Rome"], "DestCountry": ["CA"], "DestRegion": ["CA-ON"], "DestCityName": ["Ottawa"], "OriginAirportID": ["RM12"]}, "sort": [7, 7]}, {"_index": "flights", "_id": "8", "_score": null, "_source": {"FlightNum": "M05KE88", "DestCountry": "IN", "OriginWeather": "Heavy Fog", "OriginCityName": "Milan", "AvgTicketPrice": 960.8697358054351, "DistanceMiles": 4377.166776556647, "FlightDelay": true, "DestWeather": "Cloudy", "Dest": "Rajiv Gandhi International Airport", "FlightDelayType": "NAS Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 7044.367088850781, "timestamp": "2018-01-01T12:09:35", "DestLocation": {"lat": "17.23131752", "lon": "78.42985535"}, "DestAirportID": "HYD", "Carrier": "Kibana Airlines", "Cancelled": true, "FlightTimeMin": 602.0305907375651, "Origin": "Milano Linate Airport", "OriginLocation": {"lat": "45.445099", "lon": "9.27674"}, "DestRegion": "SE-BD", "OriginAirportID": "MI11", "OriginRegion": "IT-25", "DestCityName": "Hyderabad", "FlightTimeHour": 10.033843178959419, "FlightDelayMin": 15}, "fields": {"FlightNum": ["M05KE88"], "Origin": ["Milano Linate Airport"], "OriginLocation": [{"coordinates": [9.27674, 45.445099], "type": "Point"}], "DestLocation": [{"coordinates": [78.42985535, 17.23131752], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [4377.167], "FlightTimeMin": [602.0306], "OriginWeather": ["Heavy Fog"], "dayOfWeek": [0], "AvgTicketPrice": [960.86975], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [15], "OriginRegion": ["IT-25"], "DestAirportID": ["HYD"], "FlightDelayType": ["NAS Delay"], "timestamp": ["2018-01-01T12:09:35"], "Dest": ["Rajiv Gandhi International Airport"], "FlightTimeHour": [10.033843], "Cancelled": [true], "DistanceKilometers": [7044.367], "OriginCityName": ["Milan"], "DestWeather": ["Cloudy"], "OriginCountry": ["IT"], "Cities": ["Hyderabad", "Milan"], "DestCountry": ["IN"], "DestRegion": ["SE-BD"], "DestCityName": ["Hyderabad"], "OriginAirportID": ["MI11"]}, "sort": [8, 8]}, {"_index": "flights", "_id": "9", "_score": null, "_source": {"FlightNum": "SNI3M1Z", "DestCountry": "IT", "OriginWeather": "Cloudy", "OriginCityName": "Moscow", "AvgTicketPrice": 296.8777725965789, "DistanceMiles": 1303.5538675692512, "FlightDelay": false, "DestWeather": "Rain", "Dest": "Treviso-Sant\'Angelo Airport", "FlightDelayType": "No Delay", "OriginCountry": "RU", "dayOfWeek": 0, "DistanceKilometers": 2097.866595449369, "timestamp": "2018-01-01T12:09:35", "DestLocation": {"lat": "45.648399", "lon": "12.1944"}, "DestAirportID": "TV01", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 174.82221628744742, "Origin": "Sheremetyevo International Airport", "OriginLocation": {"lat": "55.972599", "lon": "37.4146"}, "DestRegion": "IT-34", "OriginAirportID": "SVO", "OriginRegion": "RU-MOS", "DestCityName": "Treviso", "FlightTimeHour": 2.91370360479079, "FlightDelayMin": 0}, "fields": {"FlightNum": ["SNI3M1Z"], "Origin": ["Sheremetyevo International Airport"], "OriginLocation": [{"coordinates": [37.4146, 55.972599], "type": "Point"}], "DestLocation": [{"coordinates": [12.1944, 45.648399], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [1303.5538], "FlightTimeMin": [174.82222], "OriginWeather": ["Cloudy"], "dayOfWeek": [0], "AvgTicketPrice": [296.87778], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["RU-MOS"], "DestAirportID": ["TV01"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T12:09:35"], "Dest": ["Treviso-Sant\'Angelo Airport"], "FlightTimeHour": [2.9137037], "Cancelled": [false], "DistanceKilometers": [2097.8667], "OriginCityName": ["Moscow"], "DestWeather": ["Rain"], "OriginCountry": ["RU"], "Cities": ["Treviso", "Moscow"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Treviso"], "OriginAirportID": ["SVO"]}, "sort": [9, 9]}, {"_index": "flights", "_id": "10", "_score": null, "_source": {"FlightNum": "JQ2XXQ5", "DestCountry": "FI", "OriginWeather": "Rain", "OriginCityName": "Albuquerque", "AvgTicketPrice": 906.4379477399872, "DistanceMiles": 5313.8222112173335, "FlightDelay": false, "DestWeather": "Rain", "Dest": "Helsinki Vantaa Airport", "FlightDelayType": "No Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers": 8551.76789268935, "timestamp": "2018-01-01T22:06:14", "DestLocation": {"lat": "60.31719971", "lon": "24.9633007"}, "DestAirportID": "HEL", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 503.04517015819704, "Origin": "Albuquerque International Sunport Airport", "OriginLocation": {"lat": "35.040199", "lon": "-106.609001"}, "DestRegion": "FI-ES", "OriginAirportID": "ABQ", "OriginRegion": "US-NM", "DestCityName": "Helsinki", "FlightTimeHour": 8.384086169303284, "FlightDelayMin": 0}, "fields": {"FlightNum": ["JQ2XXQ5"], "Origin": ["Albuquerque International Sunport Airport"], "OriginLocation": [{"coordinates": [-106.609001, 35.040199], "type": "Point"}], "DestLocation": [{"coordinates": [24.9633007, 60.31719971], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5313.8223], "FlightTimeMin": [503.04517], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [906.4379], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["US-NM"], "DestAirportID": ["HEL"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T22:06:14"], "Dest": ["Helsinki Vantaa Airport"], "FlightTimeHour": [8.384087], "Cancelled": [false], "DistanceKilometers": [8551.768], "OriginCityName": ["Albuquerque"], "DestWeather": ["Rain"], "OriginCountry": ["US"], "Cities": ["Helsinki", "Albuquerque"], "DestCountry": ["FI"], "DestRegion": ["FI-ES"], "DestCityName": ["Helsinki"], "OriginAirportID": ["ABQ"]}, "sort": [10, 10]}, {"_index": "flights", "_id": "11", "_score": null, "_source": {"FlightNum": "V30ITD0", "DestCountry": "AT", "OriginWeather": "Rain", "OriginCityName": "Venice", "AvgTicketPrice": 704.4637710312036, "DistanceMiles": 268.99172653633303, "FlightDelay": false, "DestWeather": "Cloudy", "Dest": "Vienna International Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 432.90022115088834, "timestamp": "2018-01-01T11:52:34", "DestLocation": {"lat": "48.11029816", "lon": "16.56970024"}, "DestAirportID": "VIE", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 36.07501842924069, "Origin": "Venice Marco Polo Airport", "OriginLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestRegion": "AT-9", "OriginAirportID": "VE05", "OriginRegion": "IT-34", "DestCityName": "Vienna", "FlightTimeHour": 0.6012503071540111, "FlightDelayMin": 0}, "fields": {"FlightNum": ["V30ITD0"], "Origin": ["Venice Marco Polo Airport"], "OriginLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "DestLocation": [{"coordinates": [16.56970024, 48.11029816], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [268.99173], "FlightTimeMin": [36.07502], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [704.46375], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["IT-34"], "DestAirportID": ["VIE"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T11:52:34"], "Dest": ["Vienna International Airport"], "FlightTimeHour": [0.6012503], "Cancelled": [false], "DistanceKilometers": [432.9002], "OriginCityName": ["Venice"], "DestWeather": ["Cloudy"], "OriginCountry": ["IT"], "Cities": ["Vienna", "Venice"], "DestCountry": ["AT"], "DestRegion": ["AT-9"], "DestCityName": ["Vienna"], "OriginAirportID": ["VE05"]}, "sort": [11, 11]}, {"_index": "flights", "_id": "12", "_score": null, "_source": {"FlightNum": "P0WMFH7", "DestCountry": "CN", "OriginWeather": "Heavy Fog", "OriginCityName": "Mexico City", "AvgTicketPrice": 922.499077027416, "DistanceMiles": 8025.381414737853, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Shanghai Pudong International Airport", "FlightDelayType": "No Delay", "OriginCountry": "MX", "dayOfWeek": 0, "DistanceKilometers": 12915.599427519877, "timestamp": "2018-01-01T02:13:46", "DestLocation": {"lat": "31.14340019", "lon": "121.8050003"}, "DestAirportID": "PVG", "Carrier": "Logstash Airways", "Cancelled": true, "FlightTimeMin": 679.7683909220988, "Origin": "Licenciado Benito Juarez International Airport", "OriginLocation": {"lat": "19.4363", "lon": "-99.072098"}, "DestRegion": "SE-BD", "OriginAirportID": "AICM", "OriginRegion": "MX-DIF", "DestCityName": "Shanghai", "FlightTimeHour": 11.32947318203498, "FlightDelayMin": 0}, "fields": {"FlightNum": ["P0WMFH7"], "Origin": ["Licenciado Benito Juarez International Airport"], "OriginLocation": [{"coordinates": [-99.072098, 19.4363], "type": "Point"}], "DestLocation": [{"coordinates": [121.8050003, 31.14340019], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [8025.3813], "FlightTimeMin": [679.7684], "OriginWeather": ["Heavy Fog"], "dayOfWeek": [0], "AvgTicketPrice": [922.4991], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["MX-DIF"], "DestAirportID": ["PVG"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T02:13:46"], "Dest": ["Shanghai Pudong International Airport"], "FlightTimeHour": [11.3294735], "Cancelled": [true], "DistanceKilometers": [12915.6], "OriginCityName": ["Mexico City"], "DestWeather": ["Clear"], "OriginCountry": ["MX"], "Cities": ["Shanghai", "Mexico City"], "DestCountry": ["CN"], "DestRegion": ["SE-BD"], "DestCityName": ["Shanghai"], "OriginAirportID": ["AICM"]}, "sort": [12, 12]}, {"_index": "flights", "_id": "13", "_score": null, "_source": {"FlightNum": "VT9O2KD", "DestCountry": "CA", "OriginWeather": "Rain", "OriginCityName": "Naples", "AvgTicketPrice": 374.9592762864519, "DistanceMiles": 4311.560440686985, "FlightDelay": false, "DestWeather": "Rain", "Dest": "Ottawa Macdonald-Cartier International Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 6938.783925856956, "timestamp": "2018-01-01T14:21:13", "DestLocation": {"lat": "45.32249832", "lon": "-75.66919708"}, "DestAirportID": "YOW", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 330.41828218366453, "Origin": "Naples International Airport", "OriginLocation": {"lat": "40.886002", "lon": "14.2908"}, "DestRegion": "CA-ON", "OriginAirportID": "NA01", "OriginRegion": "IT-72", "DestCityName": "Ottawa", "FlightTimeHour": 5.506971369727742, "FlightDelayMin": 0}, "fields": {"FlightNum": ["VT9O2KD"], "Origin": ["Naples International Airport"], "OriginLocation": [{"coordinates": [14.2908, 40.886002], "type": "Point"}], "DestLocation": [{"coordinates": [-75.66919708, 45.32249832], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4311.5605], "FlightTimeMin": [330.41827], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [374.9593], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["IT-72"], "DestAirportID": ["YOW"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T14:21:13"], "Dest": ["Ottawa Macdonald-Cartier International Airport"], "FlightTimeHour": [5.5069714], "Cancelled": [false], "DistanceKilometers": [6938.7837], "OriginCityName": ["Naples"], "DestWeather": ["Rain"], "OriginCountry": ["IT"], "Cities": ["Ottawa", "Naples"], "DestCountry": ["CA"], "DestRegion": ["CA-ON"], "DestCityName": ["Ottawa"], "OriginAirportID": ["NA01"]}, "sort": [13, 13]}, {"_index": "flights", "_id": "14", "_score": null, "_source": {"FlightNum": "NRHSVG8", "DestCountry": "PR", "OriginWeather": "Cloudy", "OriginCityName": "Rome", "AvgTicketPrice": 552.9173708459598, "DistanceMiles": 4806.775668847457, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Luis Munoz Marin International Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 0, "DistanceKilometers": 7735.755582005642, "timestamp": "2018-01-01T17:42:53", "DestLocation": {"lat": "18.43939972", "lon": "-66.00180054"}, "DestAirportID": "SJU", "Carrier": "Logstash Airways", "Cancelled": false, "FlightTimeMin": 407.1450306318759, "Origin": "Ciampino___G. B. Pastine International Airport", "OriginLocation": {"lat": "41.7994", "lon": "12.5949"}, "DestRegion": "PR-U-A", "OriginAirportID": "RM12", "OriginRegion": "IT-62", "DestCityName": "San Juan", "FlightTimeHour": 6.785750510531264, "FlightDelayMin": 0}, "fields": {"FlightNum": ["NRHSVG8"], "Origin": ["Ciampino___G. B. Pastine International Airport"], "OriginLocation": [{"coordinates": [12.5949, 41.7994], "type": "Point"}], "DestLocation": [{"coordinates": [-66.00180054, 18.43939972], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4806.776], "FlightTimeMin": [407.14502], "OriginWeather": ["Cloudy"], "dayOfWeek": [0], "AvgTicketPrice": [552.91736], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["IT-62"], "DestAirportID": ["SJU"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T17:42:53"], "Dest": ["Luis Munoz Marin International Airport"], "FlightTimeHour": [6.7857504], "Cancelled": [false], "DistanceKilometers": [7735.7554], "OriginCityName": ["Rome"], "DestWeather": ["Clear"], "OriginCountry": ["IT"], "Cities": ["San Juan", "Rome"], "DestCountry": ["PR"], "DestRegion": ["PR-U-A"], "DestCityName": ["San Juan"], "OriginAirportID": ["RM12"]}, "sort": [14, 14]}, {"_index": "flights", "_id": "15", "_score": null, "_source": {"FlightNum": "YIPS2BZ", "DestCountry": "DE", "OriginWeather": "Thunder & Lightning", "OriginCityName": "Chengdu", "AvgTicketPrice": 566.4875569256166, "DistanceMiles": 4896.74792596565, "FlightDelay": false, "DestWeather": "Sunny", "Dest": "Cologne Bonn Airport", "FlightDelayType": "No Delay", "OriginCountry": "CN", "dayOfWeek": 0, "DistanceKilometers": 7880.551894165264, "timestamp": "2018-01-01T19:55:32", "DestLocation": {"lat": "50.86589813", "lon": "7.142739773"}, "DestAirportID": "CGN", "Carrier": "Kibana Airlines", "Cancelled": true, "FlightTimeMin": 656.7126578471053, "Origin": "Chengdu Shuangliu International Airport", "OriginLocation": {"lat": "30.57850075", "lon": "103.9469986"}, "DestRegion": "DE-NW", "OriginAirportID": "CTU", "OriginRegion": "SE-BD", "DestCityName": "Cologne", "FlightTimeHour": 10.945210964118422, "FlightDelayMin": 0}, "fields": {"FlightNum": ["YIPS2BZ"], "Origin": ["Chengdu Shuangliu International Airport"], "OriginLocation": [{"coordinates": [103.9469986, 30.57850075], "type": "Point"}], "DestLocation": [{"coordinates": [7.142739773, 50.86589813], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4896.748], "FlightTimeMin": [656.71265], "OriginWeather": ["Thunder & Lightning"], "dayOfWeek": [0], "AvgTicketPrice": [566.48755], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["CGN"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T19:55:32"], "Dest": ["Cologne Bonn Airport"], "FlightTimeHour": [10.945211], "Cancelled": [true], "DistanceKilometers": [7880.552], "OriginCityName": ["Chengdu"], "DestWeather": ["Sunny"], "OriginCountry": ["CN"], "Cities": ["Cologne", "Chengdu"], "DestCountry": ["DE"], "DestRegion": ["DE-NW"], "DestCityName": ["Cologne"], "OriginAirportID": ["CTU"]}, "sort": [15, 15]}, {"_index": "flights", "_id": "16", "_score": null, "_source": {"FlightNum": "C7IBZ42", "DestCountry": "IT", "OriginWeather": "Thunder & Lightning", "OriginCityName": "Mexico City", "AvgTicketPrice": 989.9527866266118, "DistanceMiles": 6244.404143498341, "FlightDelay": false, "DestWeather": "Damaging Wind", "Dest": "Venice Marco Polo Airport", "FlightDelayType": "No Delay", "OriginCountry": "MX", "dayOfWeek": 0, "DistanceKilometers": 10049.394341914194, "timestamp": "2018-01-01T07:49:27", "DestLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestAirportID": "VE05", "Carrier": "Logstash Airways", "Cancelled": true, "FlightTimeMin": 773.0303339933996, "Origin": "Licenciado Benito Juarez International Airport", "OriginLocation": {"lat": "19.4363", "lon": "-99.072098"}, "DestRegion": "IT-34", "OriginAirportID": "AICM", "OriginRegion": "MX-DIF", "DestCityName": "Venice", "FlightTimeHour": 12.883838899889993, "FlightDelayMin": 0}, "fields": {"FlightNum": ["C7IBZ42"], "Origin": ["Licenciado Benito Juarez International Airport"], "OriginLocation": [{"coordinates": [-99.072098, 19.4363], "type": "Point"}], "DestLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [6244.4043], "FlightTimeMin": [773.03033], "OriginWeather": ["Thunder & Lightning"], "dayOfWeek": [0], "AvgTicketPrice": [989.95276], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["MX-DIF"], "DestAirportID": ["VE05"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T07:49:27"], "Dest": ["Venice Marco Polo Airport"], "FlightTimeHour": [12.883839], "Cancelled": [true], "DistanceKilometers": [10049.395], "OriginCityName": ["Mexico City"], "DestWeather": ["Damaging Wind"], "OriginCountry": ["MX"], "Cities": ["Venice", "Mexico City"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Venice"], "OriginAirportID": ["AICM"]}, "sort": [16, 16]}, {"_index": "flights", "_id": "17", "_score": null, "_source": {"FlightNum": "7TTZM4I", "DestCountry": "AR", "OriginWeather": "Rain", "OriginCityName": "Cleveland", "AvgTicketPrice": 569.6132552035547, "DistanceMiles": 5450.245542110189, "FlightDelay": true, "DestWeather": "Cloudy", "Dest": "Ministro Pistarini International Airport", "FlightDelayType": "NAS Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers": 8771.31996172178, "timestamp": "2018-01-01T01:30:47", "DestLocation": {"lat": "-34.8222", "lon": "-58.5358"}, "DestAirportID": "EZE", "Carrier": "ES-Air", "Cancelled": false, "FlightTimeMin": 704.7169201324446, "Origin": "Cleveland Hopkins International Airport", "OriginLocation": {"lat": "41.4117012", "lon": "-81.84980011"}, "DestRegion": "SE-BD", "OriginAirportID": "CLE", "OriginRegion": "US-OH", "DestCityName": "Buenos Aires", "FlightTimeHour": 11.745282002207409, "FlightDelayMin": 30}, "fields": {"FlightNum": ["7TTZM4I"], "Origin": ["Cleveland Hopkins International Airport"], "OriginLocation": [{"coordinates": [-81.84980011, 41.4117012], "type": "Point"}], "DestLocation": [{"coordinates": [-58.5358, -34.8222], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [5450.2456], "FlightTimeMin": [704.7169], "OriginWeather": ["Rain"], "dayOfWeek": [0], "AvgTicketPrice": [569.6133], "Carrier": ["ES-Air"], "FlightDelayMin": [30], "OriginRegion": ["US-OH"], "DestAirportID": ["EZE"], "FlightDelayType": ["NAS Delay"], "timestamp": ["2018-01-01T01:30:47"], "Dest": ["Ministro Pistarini International Airport"], "FlightTimeHour": [11.745282], "Cancelled": [false], "DistanceKilometers": [8771.32], "OriginCityName": ["Cleveland"], "DestWeather": ["Cloudy"], "OriginCountry": ["US"], "Cities": ["Buenos Aires", "Cleveland"], "DestCountry": ["AR"], "DestRegion": ["SE-BD"], "DestCityName": ["Buenos Aires"], "OriginAirportID": ["CLE"]}, "sort": [17, 17]}, {"_index": "flights", "_id": "18", "_score": null, "_source": {"FlightNum": "CSW89S3", "DestCountry": "CN", "OriginWeather": "Hail", "OriginCityName": "Olenegorsk", "AvgTicketPrice": 277.4297073844173, "DistanceMiles": 4202.458848620224, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Shanghai Pudong International Airport", "FlightDelayType": "No Delay", "OriginCountry": "RU", "dayOfWeek": 0, "DistanceKilometers": 6763.2019332738655, "timestamp": "2018-01-01T07:58:17", "DestLocation": {"lat": "31.14340019", "lon": "121.8050003"}, "DestAirportID": "PVG", "Carrier": "ES-Air", "Cancelled": false, "FlightTimeMin": 355.95799648809816, "Origin": "Olenya Air Base", "OriginLocation": {"lat": "68.15180206", "lon": "33.46390152"}, "DestRegion": "SE-BD", "OriginAirportID": "XLMO", "OriginRegion": "RU-MUR", "DestCityName": "Shanghai", "FlightTimeHour": 5.932633274801636, "FlightDelayMin": 0}, "fields": {"FlightNum": ["CSW89S3"], "Origin": ["Olenya Air Base"], "OriginLocation": [{"coordinates": [33.46390152, 68.15180206], "type": "Point"}], "DestLocation": [{"coordinates": [121.8050003, 31.14340019], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4202.459], "FlightTimeMin": [355.958], "OriginWeather": ["Hail"], "dayOfWeek": [0], "AvgTicketPrice": [277.42972], "Carrier": ["ES-Air"], "FlightDelayMin": [0], "OriginRegion": ["RU-MUR"], "DestAirportID": ["PVG"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T07:58:17"], "Dest": ["Shanghai Pudong International Airport"], "FlightTimeHour": [5.9326334], "Cancelled": [false], "DistanceKilometers": [6763.202], "OriginCityName": ["Olenegorsk"], "DestWeather": ["Clear"], "OriginCountry": ["RU"], "Cities": ["Shanghai", "Olenegorsk"], "DestCountry": ["CN"], "DestRegion": ["SE-BD"], "DestCityName": ["Shanghai"], "OriginAirportID": ["XLMO"]}, "sort": [18, 18]}, {"_index": "flights", "_id": "19", "_score": null, "_source": {"FlightNum": "RBFKZBX", "DestCountry": "IN", "OriginWeather": "Cloudy", "OriginCityName": "Casper", "AvgTicketPrice": 772.1008456460212, "DistanceMiles": 7507.304095087099, "FlightDelay": true, "DestWeather": "Clear", "Dest": "Indira Gandhi International Airport", "FlightDelayType": "Late Aircraft Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers": 12081.834801603853, "timestamp": "2018-01-01T00:02:06", "DestLocation": {"lat": "28.5665", "lon": "77.103104"}, "DestAirportID": "DEL", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 875.1146751002408, "Origin": "Casper-Natrona County International Airport", "OriginLocation": {"lat": "42.90800095", "lon": "-106.4639969"}, "DestRegion": "SE-BD", "OriginAirportID": "CPR", "OriginRegion": "US-WY", "DestCityName": "New Delhi", "FlightTimeHour": 14.585244585004013, "FlightDelayMin": 120}, "fields": {"FlightNum": ["RBFKZBX"], "Origin": ["Casper-Natrona County International Airport"], "OriginLocation": [{"coordinates": [-106.4639969, 42.90800095], "type": "Point"}], "DestLocation": [{"coordinates": [77.103104, 28.5665], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [7507.304], "FlightTimeMin": [875.1147], "OriginWeather": ["Cloudy"], "dayOfWeek": [0], "AvgTicketPrice": [772.1008], "Carrier": ["JetBeats"], "FlightDelayMin": [120], "OriginRegion": ["US-WY"], "DestAirportID": ["DEL"], "FlightDelayType": ["Late Aircraft Delay"], "timestamp": ["2018-01-01T00:02:06"], "Dest": ["Indira Gandhi International Airport"], "FlightTimeHour": [14.585244], "Cancelled": [false], "DistanceKilometers": [12081.835], "OriginCityName": ["Casper"], "DestWeather": ["Clear"], "OriginCountry": ["US"], "Cities": ["New Delhi", "Casper"], "DestCountry": ["IN"], "DestRegion": ["SE-BD"], "DestCityName": ["New Delhi"], "OriginAirportID": ["CPR"]}, "sort": [19, 19]}, {"_index": "flights", "_id": "20", "_score": null, "_source": {"FlightNum": "R43CELD", "DestCountry": "US", "OriginWeather": "Cloudy", "OriginCityName": "Erie", "AvgTicketPrice": 167.59992219374266, "DistanceMiles": 965.1786928006221, "FlightDelay": true, "DestWeather": "Clear", "Dest": "Wichita Mid Continent Airport", "FlightDelayType": "Carrier Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers": 1553.3045381865245, "timestamp": "2018-01-01T01:08:20", "DestLocation": {"lat": "37.64989853", "lon": "-97.43309784"}, "DestAirportID": "ICT", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 373.96688277078687, "Origin": "Erie International Tom Ridge Field", "OriginLocation": {"lat": "42.08312701", "lon": "-80.17386675"}, "DestRegion": "US-KS", "OriginAirportID": "ERI", "OriginRegion": "US-PA", "DestCityName": "Wichita", "FlightTimeHour": 6.232781379513114, "FlightDelayMin": 300}, "fields": {"FlightNum": ["R43CELD"], "Origin": ["Erie International Tom Ridge Field"], "OriginLocation": [{"coordinates": [-80.17386675, 42.08312701], "type": "Point"}], "DestLocation": [{"coordinates": [-97.43309784, 37.64989853], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [965.1787], "FlightTimeMin": [373.9669], "OriginWeather": ["Cloudy"], "dayOfWeek": [0], "AvgTicketPrice": [167.59993], "Carrier": ["JetBeats"], "FlightDelayMin": [300], "OriginRegion": ["US-PA"], "DestAirportID": ["ICT"], "FlightDelayType": ["Carrier Delay"], "timestamp": ["2018-01-01T01:08:20"], "Dest": ["Wichita Mid Continent Airport"], "FlightTimeHour": [6.2327814], "Cancelled": [false], "DistanceKilometers": [1553.3046], "OriginCityName": ["Erie"], "DestWeather": ["Clear"], "OriginCountry": ["US"], "Cities": ["Wichita", "Erie"], "DestCountry": ["US"], "DestRegion": ["US-KS"], "DestCityName": ["Wichita"], "OriginAirportID": ["ERI"]}, "sort": [20, 20]}, {"_index": "flights", "_id": "21", "_score": null, "_source": {"FlightNum": "1TJWK8F", "DestCountry": "CA", "OriginWeather": "Clear", "OriginCityName": "Newark", "AvgTicketPrice": 253.21006490337038, "DistanceMiles": 328.5065863946441, "FlightDelay": true, "DestWeather": "Hail", "Dest": "Ottawa Macdonald-Cartier International Airport", "FlightDelayType": "NAS Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers": 528.6801037747022, "timestamp": "2018-01-01T01:08:20", "DestLocation": {"lat": "45.32249832", "lon": "-75.66919708"}, "DestAirportID": "YOW", "Carrier": "ES-Air", "Cancelled": false, "FlightTimeMin": 130.66770029036172, "Origin": "Newark Liberty International Airport", "OriginLocation": {"lat": "40.69250107", "lon": "-74.16870117"}, "DestRegion": "CA-ON", "OriginAirportID": "EWR", "OriginRegion": "US-NJ", "DestCityName": "Ottawa", "FlightTimeHour": 2.177795004839362, "FlightDelayMin": 90}, "fields": {"FlightNum": ["1TJWK8F"], "Origin": ["Newark Liberty International Airport"], "OriginLocation": [{"coordinates": [-74.16870117, 40.69250107], "type": "Point"}], "DestLocation": [{"coordinates": [-75.66919708, 45.32249832], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [328.5066], "FlightTimeMin": [130.6677], "OriginWeather": ["Clear"], "dayOfWeek": [0], "AvgTicketPrice": [253.21007], "Carrier": ["ES-Air"], "FlightDelayMin": [90], "OriginRegion": ["US-NJ"], "DestAirportID": ["YOW"], "FlightDelayType": ["NAS Delay"], "timestamp": ["2018-01-01T01:08:20"], "Dest": ["Ottawa Macdonald-Cartier International Airport"], "FlightTimeHour": [2.177795], "Cancelled": [false], "DistanceKilometers": [528.6801], "OriginCityName": ["Newark"], "DestWeather": ["Hail"], "OriginCountry": ["US"], "Cities": ["Ottawa", "Newark"], "DestCountry": ["CA"], "DestRegion": ["CA-ON"], "DestCityName": ["Ottawa"], "OriginAirportID": ["EWR"]}, "sort": [21, 21]}, {"_index": "flights", "_id": "22", "_score": null, "_source": {"FlightNum": "UDTSN13", "DestCountry": "JP", "OriginWeather": "Sunny", "OriginCityName": "Copenhagen", "AvgTicketPrice": 917.2476203642742, "DistanceMiles": 5354.622537746889, "FlightDelay": false, "DestWeather": "Damaging Wind", "Dest": "Itami Airport", "FlightDelayType": "No Delay", "OriginCountry": "DK", "dayOfWeek": 0, "DistanceKilometers": 8617.42965338773, "timestamp": "2018-01-01T07:48:35", "DestLocation": {"lat": "34.78549957", "lon": "135.4380035"}, "DestAirportID": "ITM", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 574.4953102258486, "Origin": "Copenhagen Kastrup Airport", "OriginLocation": {"lat": "55.61790085", "lon": "12.65600014"}, "DestRegion": "SE-BD", "OriginAirportID": "CPH", "OriginRegion": "DK-84", "DestCityName": "Osaka", "FlightTimeHour": 9.574921837097476, "FlightDelayMin": 0}, "fields": {"FlightNum": ["UDTSN13"], "Origin": ["Copenhagen Kastrup Airport"], "OriginLocation": [{"coordinates": [12.65600014, 55.61790085], "type": "Point"}], "DestLocation": [{"coordinates": [135.4380035, 34.78549957], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5354.6226], "FlightTimeMin": [574.4953], "OriginWeather": ["Sunny"], "dayOfWeek": [0], "AvgTicketPrice": [917.2476], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["DK-84"], "DestAirportID": ["ITM"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-01T07:48:35"], "Dest": ["Itami Airport"], "FlightTimeHour": [9.574922], "Cancelled": [false], "DistanceKilometers": [8617.43], "OriginCityName": ["Copenhagen"], "DestWeather": ["Damaging Wind"], "OriginCountry": ["DK"], "Cities": ["Osaka", "Copenhagen"], "DestCountry": ["JP"], "DestRegion": ["SE-BD"], "DestCityName": ["Osaka"], "OriginAirportID": ["CPH"]}, "sort": [22, 22]}, {"_index": "flights", "_id": "23", "_score": null, "_source": {"FlightNum": "7Y08OAU", "DestCountry": "AT", "OriginWeather": "Heavy Fog", "OriginCityName": "Seattle", "AvgTicketPrice": 451.5911757026697, "DistanceMiles": 5403.40296642641, "FlightDelay": false, "DestWeather": "Heavy Fog", "Dest": "Vienna International Airport", "FlightDelayType": "No Delay", "OriginCountry": "US", "dayOfWeek": 0, "DistanceKilometers...iginRegion": "SE-BD", "DestCityName": "Sydney", "FlightTimeHour": 9.368951522261131, "FlightDelayMin": 0}, "fields": {"FlightNum": ["W6GJ06R"], "Origin": ["Rajiv Gandhi International Airport"], "OriginLocation": [{"coordinates": [78.42985535, 17.23131752], "type": "Point"}], "DestLocation": [{"coordinates": [151.177002, -33.94609833], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5938.0283], "FlightTimeMin": [562.1371], "OriginWeather": ["Sunny"], "dayOfWeek": [1], "AvgTicketPrice": [606.7856], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["SYD"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T00:07:20"], "Dest": ["Sydney Kingsford Smith International Airport"], "FlightTimeHour": [9.368952], "Cancelled": [false], "DistanceKilometers": [9556.33], "OriginCityName": ["Hyderabad"], "DestWeather": ["Rain"], "OriginCountry": ["IN"], "Cities": ["Sydney", "Hyderabad"], "DestCountry": ["AU"], "DestRegion": ["SE-BD"], "DestCityName": ["Sydney"], "OriginAirportID": ["HYD"]}, "sort": [4988, 4988]}, {"_index": "flights", "_id": "4989", "_score": null, "_source": {"FlightNum": "501NGTG", "DestCountry": "US", "OriginWeather": "Rain", "OriginCityName": "Abu Dhabi", "AvgTicketPrice": 682.4858777594711, "DistanceMiles": 6867.0238729545945, "FlightDelay": false, "DestWeather": "Cloudy", "Dest": "John F Kennedy International Airport", "FlightDelayType": "No Delay", "OriginCountry": "AE", "dayOfWeek": 1, "DistanceKilometers": 11051.40366779624, "timestamp": "2018-01-16T16:04:32", "DestLocation": {"lat": "40.63980103", "lon": "-73.77890015"}, "DestAirportID": "JFK", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 736.7602445197493, "Origin": "Abu Dhabi International Airport", "OriginLocation": {"lat": "24.43300056", "lon": "54.65110016"}, "DestRegion": "US-NY", "OriginAirportID": "AUH", "OriginRegion": "SE-BD", "DestCityName": "New York", "FlightTimeHour": 12.279337408662489, "FlightDelayMin": 0}, "fields": {"FlightNum": ["501NGTG"], "Origin": ["Abu Dhabi International Airport"], "OriginLocation": [{"coordinates": [54.65110016, 24.43300056], "type": "Point"}], "DestLocation": [{"coordinates": [-73.77890015, 40.63980103], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [6867.024], "FlightTimeMin": [736.76025], "OriginWeather": ["Rain"], "dayOfWeek": [1], "AvgTicketPrice": [682.4859], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["JFK"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T16:04:32"], "Dest": ["John F Kennedy International Airport"], "FlightTimeHour": [12.279338], "Cancelled": [false], "DistanceKilometers": [11051.403], "OriginCityName": ["Abu Dhabi"], "DestWeather": ["Cloudy"], "OriginCountry": ["AE"], "Cities": ["New York", "Abu Dhabi"], "DestCountry": ["US"], "DestRegion": ["US-NY"], "DestCityName": ["New York"], "OriginAirportID": ["AUH"]}, "sort": [4989, 4989]}, {"_index": "flights", "_id": "4990", "_score": null, "_source": {"FlightNum": "X2SBMJY", "DestCountry": "IT", "OriginWeather": "Sunny", "OriginCityName": "Buenos Aires", "AvgTicketPrice": 883.1166760108651, "DistanceMiles": 7078.994713303113, "FlightDelay": false, "DestWeather": "Sunny", "Dest": "Treviso-Sant\'Angelo Airport", "FlightDelayType": "No Delay", "OriginCountry": "AR", "dayOfWeek": 1, "DistanceKilometers": 11392.537667886085, "timestamp": "2018-01-16T23:51:54", "DestLocation": {"lat": "45.648399", "lon": "12.1944"}, "DestAirportID": "TV01", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 670.1492745815344, "Origin": "Ministro Pistarini International Airport", "OriginLocation": {"lat": "-34.8222", "lon": "-58.5358"}, "DestRegion": "IT-34", "OriginAirportID": "EZE", "OriginRegion": "SE-BD", "DestCityName": "Treviso", "FlightTimeHour": 11.169154576358908, "FlightDelayMin": 0}, "fields": {"FlightNum": ["X2SBMJY"], "Origin": ["Ministro Pistarini International Airport"], "OriginLocation": [{"coordinates": [-58.5358, -34.8222], "type": "Point"}], "DestLocation": [{"coordinates": [12.1944, 45.648399], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [7078.9946], "FlightTimeMin": [670.1493], "OriginWeather": ["Sunny"], "dayOfWeek": [1], "AvgTicketPrice": [883.1167], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["TV01"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T23:51:54"], "Dest": ["Treviso-Sant\'Angelo Airport"], "FlightTimeHour": [11.169154], "Cancelled": [false], "DistanceKilometers": [11392.538], "OriginCityName": ["Buenos Aires"], "DestWeather": ["Sunny"], "OriginCountry": ["AR"], "Cities": ["Treviso", "Buenos Aires"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Treviso"], "OriginAirportID": ["EZE"]}, "sort": [4990, 4990]}, {"_index": "flights", "_id": "4991", "_score": null, "_source": {"FlightNum": "66MVCEQ", "DestCountry": "US", "OriginWeather": "Rain", "OriginCityName": "Quito", "AvgTicketPrice": 466.0494531822727, "DistanceMiles": 3073.7840534488773, "FlightDelay": false, "DestWeather": "Heavy Fog", "Dest": "Tucson International Airport", "FlightDelayType": "No Delay", "OriginCountry": "EC", "dayOfWeek": 1, "DistanceKilometers": 4946.775923713631, "timestamp": "2018-01-16T16:38:34", "DestLocation": {"lat": "32.11610031", "lon": "-110.9410019"}, "DestAirportID": "TUS", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 412.2313269761359, "Origin": "Mariscal Sucre International Airport", "OriginLocation": {"lat": "-0.129166667", "lon": "-78.3575"}, "DestRegion": "US-AZ", "OriginAirportID": "UIO", "OriginRegion": "SE-BD", "DestCityName": "Tucson", "FlightTimeHour": 6.870522116268932, "FlightDelayMin": 0}, "fields": {"FlightNum": ["66MVCEQ"], "Origin": ["Mariscal Sucre International Airport"], "OriginLocation": [{"coordinates": [-78.3575, -0.129166667], "type": "Point"}], "DestLocation": [{"coordinates": [-110.9410019, 32.11610031], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [3073.784], "FlightTimeMin": [412.23132], "OriginWeather": ["Rain"], "dayOfWeek": [1], "AvgTicketPrice": [466.04944], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["TUS"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T16:38:34"], "Dest": ["Tucson International Airport"], "FlightTimeHour": [6.870522], "Cancelled": [false], "DistanceKilometers": [4946.776], "OriginCityName": ["Quito"], "DestWeather": ["Heavy Fog"], "OriginCountry": ["EC"], "Cities": ["Tucson", "Quito"], "DestCountry": ["US"], "DestRegion": ["US-AZ"], "DestCityName": ["Tucson"], "OriginAirportID": ["UIO"]}, "sort": [4991, 4991]}, {"_index": "flights", "_id": "4992", "_score": null, "_source": {"FlightNum": "8S0B3D8", "DestCountry": "US", "OriginWeather": "Clear", "OriginCityName": "Santiago", "AvgTicketPrice": 796.1881759719555, "DistanceMiles": 5179.6277456207545, "FlightDelay": false, "DestWeather": "Sunny", "Dest": "Wichita Mid Continent Airport", "FlightDelayType": "No Delay", "OriginCountry": "CL", "dayOfWeek": 1, "DistanceKilometers": 8335.802834648288, "timestamp": "2018-01-16T04:21:46", "DestLocation": {"lat": "37.64989853", "lon": "-97.43309784"}, "DestAirportID": "ICT", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 595.4144881891634, "Origin": "Comodoro Arturo Merino Benitez International Airport", "OriginLocation": {"lat": "-33.39300156", "lon": "-70.78579712"}, "DestRegion": "US-KS", "OriginAirportID": "SCL", "OriginRegion": "SE-BD", "DestCityName": "Wichita", "FlightTimeHour": 9.923574803152723, "FlightDelayMin": 0}, "fields": {"FlightNum": ["8S0B3D8"], "Origin": ["Comodoro Arturo Merino Benitez International Airport"], "OriginLocation": [{"coordinates": [-70.78579712, -33.39300156], "type": "Point"}], "DestLocation": [{"coordinates": [-97.43309784, 37.64989853], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5179.628], "FlightTimeMin": [595.4145], "OriginWeather": ["Clear"], "dayOfWeek": [1], "AvgTicketPrice": [796.1882], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["ICT"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T04:21:46"], "Dest": ["Wichita Mid Continent Airport"], "FlightTimeHour": [9.923574], "Cancelled": [false], "DistanceKilometers": [8335.803], "OriginCityName": ["Santiago"], "DestWeather": ["Sunny"], "OriginCountry": ["CL"], "Cities": ["Wichita", "Santiago"], "DestCountry": ["US"], "DestRegion": ["US-KS"], "DestCityName": ["Wichita"], "OriginAirportID": ["SCL"]}, "sort": [4992, 4992]}, {"_index": "flights", "_id": "4993", "_score": null, "_source": {"FlightNum": "MSZV5S1", "DestCountry": "IT", "OriginWeather": "Rain", "OriginCityName": "Bergamo", "AvgTicketPrice": 237.8535010114335, "DistanceMiles": 128.90429317169247, "FlightDelay": false, "DestWeather": "Rain", "Dest": "Venice Marco Polo Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 1, "DistanceKilometers": 207.45135079010424, "timestamp": "2018-01-16T02:13:43", "DestLocation": {"lat": "45.505299", "lon": "12.3519"}, "DestAirportID": "VE05", "Carrier": "ES-Air", "Cancelled": false, "FlightTimeMin": 10.918492146847592, "Origin": "Il Caravaggio International Airport", "OriginLocation": {"lat": "45.673901", "lon": "9.70417"}, "DestRegion": "IT-34", "OriginAirportID": "BG01", "OriginRegion": "IT-25", "DestCityName": "Venice", "FlightTimeHour": 0.18197486911412603, "FlightDelayMin": 0}, "fields": {"FlightNum": ["MSZV5S1"], "Origin": ["Il Caravaggio International Airport"], "OriginLocation": [{"coordinates": [9.70417, 45.673901], "type": "Point"}], "DestLocation": [{"coordinates": [12.3519, 45.505299], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [128.9043], "FlightTimeMin": [10.918492], "OriginWeather": ["Rain"], "dayOfWeek": [1], "AvgTicketPrice": [237.8535], "Carrier": ["ES-Air"], "FlightDelayMin": [0], "OriginRegion": ["IT-25"], "DestAirportID": ["VE05"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T02:13:43"], "Dest": ["Venice Marco Polo Airport"], "FlightTimeHour": [0.18197487], "Cancelled": [false], "DistanceKilometers": [207.45135], "OriginCityName": ["Bergamo"], "DestWeather": ["Rain"], "OriginCountry": ["IT"], "Cities": ["Venice", "Bergamo"], "DestCountry": ["IT"], "DestRegion": ["IT-34"], "DestCityName": ["Venice"], "OriginAirportID": ["BG01"]}, "sort": [4993, 4993]}, {"_index": "flights", "_id": "4994", "_score": null, "_source": {"FlightNum": "JYAFU5N", "DestCountry": "DK", "OriginWeather": "Sunny", "OriginCityName": "Green Bay", "AvgTicketPrice": 633.2095635501828, "DistanceMiles": 4144.532208591916, "FlightDelay": false, "DestWeather": "Clear", "Dest": "Copenhagen Kastrup Airport", "FlightDelayType": "No Delay", "OriginCountry": "US", "dayOfWeek": 1, "DistanceKilometers": 6669.97804270415, "timestamp": "2018-01-16T10:25:12", "DestLocation": {"lat": "55.61790085", "lon": "12.65600014"}, "DestAirportID": "CPH", "Carrier": "Kibana Airlines", "Cancelled": false, "FlightTimeMin": 370.5543357057861, "Origin": "Austin Straubel International Airport", "OriginLocation": {"lat": "44.48509979", "lon": "-88.12960052"}, "DestRegion": "DK-84", "OriginAirportID": "GRB", "OriginRegion": "US-WI", "DestCityName": "Copenhagen", "FlightTimeHour": 6.175905595096435, "FlightDelayMin": 0}, "fields": {"FlightNum": ["JYAFU5N"], "Origin": ["Austin Straubel International Airport"], "OriginLocation": [{"coordinates": [-88.12960052, 44.48509979], "type": "Point"}], "DestLocation": [{"coordinates": [12.65600014, 55.61790085], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4144.532], "FlightTimeMin": [370.55432], "OriginWeather": ["Sunny"], "dayOfWeek": [1], "AvgTicketPrice": [633.20953], "Carrier": ["Kibana Airlines"], "FlightDelayMin": [0], "OriginRegion": ["US-WI"], "DestAirportID": ["CPH"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T10:25:12"], "Dest": ["Copenhagen Kastrup Airport"], "FlightTimeHour": [6.1759057], "Cancelled": [false], "DistanceKilometers": [6669.978], "OriginCityName": ["Green Bay"], "DestWeather": ["Clear"], "OriginCountry": ["US"], "Cities": ["Copenhagen", "Green Bay"], "DestCountry": ["DK"], "DestRegion": ["DK-84"], "DestCityName": ["Copenhagen"], "OriginAirportID": ["GRB"]}, "sort": [4994, 4994]}, {"_index": "flights", "_id": "4995", "_score": null, "_source": {"FlightNum": "XNPCN26", "DestCountry": "US", "OriginWeather": "Cloudy", "OriginCityName": "Chitose / Tomakomai", "AvgTicketPrice": 311.39713291763474, "DistanceMiles": 5077.386181734518, "FlightDelay": true, "DestWeather": "Thunder & Lightning", "Dest": "Salt Lake City International Airport", "FlightDelayType": "Carrier Delay", "OriginCountry": "JP", "dayOfWeek": 1, "DistanceKilometers": 8171.260987257357, "timestamp": "2018-01-16T15:03:43", "DestLocation": {"lat": "40.78839874", "lon": "-111.9779968"}, "DestAirportID": "SLC", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 937.841907932487, "Origin": "New Chitose Airport", "OriginLocation": {"lat": "42.77519989", "lon": "141.6920013"}, "DestRegion": "US-UT", "OriginAirportID": "CTS", "OriginRegion": "SE-BD", "DestCityName": "Salt Lake City", "FlightTimeHour": 15.63069846554145, "FlightDelayMin": 195}, "fields": {"FlightNum": ["XNPCN26"], "Origin": ["New Chitose Airport"], "OriginLocation": [{"coordinates": [141.6920013, 42.77519989], "type": "Point"}], "DestLocation": [{"coordinates": [-111.9779968, 40.78839874], "type": "Point"}], "FlightDelay": [true], "DistanceMiles": [5077.386], "FlightTimeMin": [937.8419], "OriginWeather": ["Cloudy"], "dayOfWeek": [1], "AvgTicketPrice": [311.39713], "Carrier": ["JetBeats"], "FlightDelayMin": [195], "OriginRegion": ["SE-BD"], "DestAirportID": ["SLC"], "FlightDelayType": ["Carrier Delay"], "timestamp": ["2018-01-16T15:03:43"], "Dest": ["Salt Lake City International Airport"], "FlightTimeHour": [15.630698], "Cancelled": [false], "DistanceKilometers": [8171.261], "OriginCityName": ["Chitose / Tomakomai"], "DestWeather": ["Thunder & Lightning"], "OriginCountry": ["JP"], "Cities": ["Salt Lake City", "Chitose / Tomakomai"], "DestCountry": ["US"], "DestRegion": ["US-UT"], "DestCityName": ["Salt Lake City"], "OriginAirportID": ["CTS"]}, "sort": [4995, 4995]}, {"_index": "flights", "_id": "4996", "_score": null, "_source": {"FlightNum": "YP127FP", "DestCountry": "US", "OriginWeather": "Damaging Wind", "OriginCityName": "Rome", "AvgTicketPrice": 618.0238956066455, "DistanceMiles": 5400.13654607216, "FlightDelay": false, "DestWeather": "Sunny", "Dest": "Wichita Mid Continent Airport", "FlightDelayType": "No Delay", "OriginCountry": "IT", "dayOfWeek": 1, "DistanceKilometers": 8690.677349601954, "timestamp": "2018-01-16T08:47:56", "DestLocation": {"lat": "37.64989853", "lon": "-97.43309784"}, "DestAirportID": "ICT", "Carrier": "Logstash Airways", "Cancelled": true, "FlightTimeMin": 457.4040710316818, "Origin": "Leonardo da Vinci - Fiumicino Airport", "OriginLocation": {"lat": "41.8002778", "lon": "12.2388889"}, "DestRegion": "US-KS", "OriginAirportID": "FCO", "OriginRegion": "SE-BD", "DestCityName": "Wichita", "FlightTimeHour": 7.623401183861363, "FlightDelayMin": 0}, "fields": {"FlightNum": ["YP127FP"], "Origin": ["Leonardo da Vinci - Fiumicino Airport"], "OriginLocation": [{"coordinates": [12.2388889, 41.8002778], "type": "Point"}], "DestLocation": [{"coordinates": [-97.43309784, 37.64989853], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [5400.1367], "FlightTimeMin": [457.40408], "OriginWeather": ["Damaging Wind"], "dayOfWeek": [1], "AvgTicketPrice": [618.0239], "Carrier": ["Logstash Airways"], "FlightDelayMin": [0], "OriginRegion": ["SE-BD"], "DestAirportID": ["ICT"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T08:47:56"], "Dest": ["Wichita Mid Continent Airport"], "FlightTimeHour": [7.623401], "Cancelled": [true], "DistanceKilometers": [8690.678], "OriginCityName": ["Rome"], "DestWeather": ["Sunny"], "OriginCountry": ["IT"], "Cities": ["Wichita", "Rome"], "DestCountry": ["US"], "DestRegion": ["US-KS"], "DestCityName": ["Wichita"], "OriginAirportID": ["FCO"]}, "sort": [4996, 4996]}, {"_index": "flights", "_id": "4997", "_score": null, "_source": {"FlightNum": "KGQK1B4", "DestCountry": "CA", "OriginWeather": "Rain", "OriginCityName": "Warsaw", "AvgTicketPrice": 730.4717239616782, "DistanceMiles": 4527.391333277357, "FlightDelay": false, "DestWeather": "Hail", "Dest": "Winnipeg / James Armstrong Richardson International Airport", "FlightDelayType": "No Delay", "OriginCountry": "PL", "dayOfWeek": 1, "DistanceKilometers": 7286.130077861916, "timestamp": "2018-01-16T20:20:17", "DestLocation": {"lat": "49.90999985", "lon": "-97.23989868"}, "DestAirportID": "YWG", "Carrier": "ES-Air", "Cancelled": true, "FlightTimeMin": 455.38312986636976, "Origin": "Warsaw Chopin Airport", "OriginLocation": {"lat": "52.16569901", "lon": "20.96710014"}, "DestRegion": "CA-MB", "OriginAirportID": "WAW", "OriginRegion": "PL-MZ", "DestCityName": "Winnipeg", "FlightTimeHour": 7.589718831106163, "FlightDelayMin": 0}, "fields": {"FlightNum": ["KGQK1B4"], "Origin": ["Warsaw Chopin Airport"], "OriginLocation": [{"coordinates": [20.96710014, 52.16569901], "type": "Point"}], "DestLocation": [{"coordinates": [-97.23989868, 49.90999985], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4527.391], "FlightTimeMin": [455.38312], "OriginWeather": ["Rain"], "dayOfWeek": [1], "AvgTicketPrice": [730.47174], "Carrier": ["ES-Air"], "FlightDelayMin": [0], "OriginRegion": ["PL-MZ"], "DestAirportID": ["YWG"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T20:20:17"], "Dest": ["Winnipeg / James Armstrong Richardson International Airport"], "FlightTimeHour": [7.589719], "Cancelled": [true], "DistanceKilometers": [7286.13], "OriginCityName": ["Warsaw"], "DestWeather": ["Hail"], "OriginCountry": ["PL"], "Cities": ["Winnipeg", "Warsaw"], "DestCountry": ["CA"], "DestRegion": ["CA-MB"], "DestCityName": ["Winnipeg"], "OriginAirportID": ["WAW"]}, "sort": [4997, 4997]}, {"_index": "flights", "_id": "4998", "_score": null, "_source": {"FlightNum": "VU9HDBO", "DestCountry": "PL", "OriginWeather": "Heavy Fog", "OriginCityName": "Belogorsk", "AvgTicketPrice": 879.9827336371554, "DistanceMiles": 4159.88094808335, "FlightDelay": false, "DestWeather": "Hail", "Dest": "Warsaw Chopin Airport", "FlightDelayType": "No Delay", "OriginCountry": "RU", "dayOfWeek": 1, "DistanceKilometers": 6694.679444512251, "timestamp": "2018-01-16T22:43:58", "DestLocation": {"lat": "52.16569901", "lon": "20.96710014"}, "DestAirportID": "WAW", "Carrier": "ES-Air", "Cancelled": false, "FlightTimeMin": 418.4174652820157, "Origin": "Ukrainka Air Base", "OriginLocation": {"lat": "51.169997", "lon": "128.445007"}, "DestRegion": "PL-MZ", "OriginAirportID": "XHBU", "OriginRegion": "RU-AMU", "DestCityName": "Warsaw", "FlightTimeHour": 6.973624421366929, "FlightDelayMin": 0}, "fields": {"FlightNum": ["VU9HDBO"], "Origin": ["Ukrainka Air Base"], "OriginLocation": [{"coordinates": [128.445007, 51.169997], "type": "Point"}], "DestLocation": [{"coordinates": [20.96710014, 52.16569901], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [4159.881], "FlightTimeMin": [418.41748], "OriginWeather": ["Heavy Fog"], "dayOfWeek": [1], "AvgTicketPrice": [879.9827], "Carrier": ["ES-Air"], "FlightDelayMin": [0], "OriginRegion": ["RU-AMU"], "DestAirportID": ["WAW"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T22:43:58"], "Dest": ["Warsaw Chopin Airport"], "FlightTimeHour": [6.973624], "Cancelled": [false], "DistanceKilometers": [6694.6797], "OriginCityName": ["Belogorsk"], "DestWeather": ["Hail"], "OriginCountry": ["RU"], "Cities": ["Warsaw", "Belogorsk"], "DestCountry": ["PL"], "DestRegion": ["PL-MZ"], "DestCityName": ["Warsaw"], "OriginAirportID": ["XHBU"]}, "sort": [4998, 4998]}, {"_index": "flights", "_id": "4999", "_score": null, "_source": {"FlightNum": "P5F9MCZ", "DestCountry": "MX", "OriginWeather": "Rain", "OriginCityName": "Edmonton", "AvgTicketPrice": 299.94926749106776, "DistanceMiles": 2461.2994214866576, "FlightDelay": false, "DestWeather": "Thunder & Lightning", "Dest": "Licenciado Benito Juarez International Airport", "FlightDelayType": "No Delay", "OriginCountry": "CA", "dayOfWeek": 1, "DistanceKilometers": 3961.0774561730236, "timestamp": "2018-01-16T07:35:22", "DestLocation": {"lat": "19.4363", "lon": "-99.072098"}, "DestAirportID": "AICM", "Carrier": "JetBeats", "Cancelled": false, "FlightTimeMin": 360.09795056118395, "Origin": "Edmonton International Airport", "OriginLocation": {"lat": "53.30970001", "lon": "-113.5800018"}, "DestRegion": "MX-DIF", "OriginAirportID": "CYEG", "OriginRegion": "CA-AB", "DestCityName": "Mexico City", "FlightTimeHour": 6.001632509353066, "FlightDelayMin": 0}, "fields": {"FlightNum": ["P5F9MCZ"], "Origin": ["Edmonton International Airport"], "OriginLocation": [{"coordinates": [-113.5800018, 53.30970001], "type": "Point"}], "DestLocation": [{"coordinates": [-99.072098, 19.4363], "type": "Point"}], "FlightDelay": [false], "DistanceMiles": [2461.2993], "FlightTimeMin": [360.09796], "OriginWeather": ["Rain"], "dayOfWeek": [1], "AvgTicketPrice": [299.94928], "Carrier": ["JetBeats"], "FlightDelayMin": [0], "OriginRegion": ["CA-AB"], "DestAirportID": ["AICM"], "FlightDelayType": ["No Delay"], "timestamp": ["2018-01-16T07:35:22"], "Dest": ["Licenciado Benito Juarez International Airport"], "FlightTimeHour": [6.0016327], "Cancelled": [false], "DistanceKilometers": [3961.0774], "OriginCityName": ["Edmonton"], "DestWeather": ["Thunder & Lightning"], "OriginCountry": ["CA"], "Cities": ["Mexico City", "Edmonton"], "DestCountry": ["MX"], "DestRegion": ["MX-DIF"], "DestCityName": ["Mexico City"], "OriginAirportID": ["CYEG"]}, "sort": [4999, 4999]}]}}

This actually shows a lot of duplicate data going over the wire. Perhaps it's possible to come up with what field names to put in _source and what field names need to go in fields, before sending the request.

The data was obtained by setting a breakpoint on line 1618 in eland/operations.py.

[pquentin]

Thanks! LGTM. It seems a bit hacky but I don't have better ideas.