Dataset Documentation

Spatial Language OpenStreetMap Dataset (SL-OSM)

Subdirectories

• amt: processed AMT responses for faunce, dorrance and the other 5 maps.

• frame_of_ref: blind and not blind frame of reference annotations. The processed data (containing the entries in the AMT csv file plus annotated FoR) are in sg_processed/*-positive.pkl

• mapinfo: Contains many .json files plus map image about different information of the maps grouped by cities. A convenient class “MapInfoDataset” can be used to quickly load these map info

• sg_parsed: The spatial graphs (i.e. collection of TargetSymbol, relation, Landmark tuples) along with other info (e.g. raw language map name, target locations etc). Each sg-*.json is basically a object search problem setting that can be executed.

• sg_parsed_annotated: We randomly chose ~20 languages per city and manually annotated the tuples. Each sg-*.json here has the same format as above and still specifies an object search case (the only difference is the parsing is done manually).

• sg_parsed_annotated_seed_matching: This is identical to sg_parsed_annotated except that the seeds (i.e. the integer at the end of the JSON file for each spatial graph) correspond to the trials in the experiment results.

Documentation

amt

There are two .csv files, which are collected using AMT. The turker is asked to provide a spatial language description for the locations of two target objects (green car, red car, or red bike) with respect to landmarks on the map.

• amt_fau_dor_data.csv: Spatial language descriptions corresponding to Faunce and Dorrance, two locations in Providence, RI. This was an early pilot data collection.

• amt_output_data_jul_merged.csv: Spatial language descriptions corresponding to areas in five cities (Austin, Cleveland, Denver, Honolulu, Washington DC). This data was later processed and used in our experiments.

  Fields:

  • map_name: Name of city for the map
  • map_num: Used to index screenshots for target object configurations
  • obj1_name: Name of first object (rcar, gcar or bike)
  • obj2_name: Name of second object (rcar, gcar or bike)
  • hint: Spatial langauge description
  • map_link: Link to the image file on Google corresponding to this configuration
  • rcar_loc: Location (an index) of red car (integer, -1 if not on the map).
  • bike_loc: Location (an index) of red bike (integer, -1 if not on the map).
  • gcar_loc: Location (an index) of green car (integer, -1 if not on the map).

  Note that you can use the idx_to_pomdp function in MapInfoDataset to convert the integer rcar_loc, bike_loc, or gcar_loc into 2D grid X, Y coordinates.

sg_parsed

For each row in the AMT CSV file, we parsed the spatial language and store the result, as well as data from other fields into a .json file. The prefix sg means "spatial graph", which is essentially just a collection of (figure, relation, ground) tuples, mentioned in the paper, that corresponds to a spatial language observation.

For example, in sg-1.json:

{
    "bike_loc": "-1",
    "entities": [
        "TheYork",
        "RedHonda",
        "GreenToyota",
        "StatesmanApartments",
        "EmpireApartments"
    ],
    "gcar_loc": "929",
    "index": 1,
    "lang": "The GreenToyota is with with west of EmpireApartments and with north of StatesmanApartments . the RedHonda is with south of StatesmanApartments and with with west of TheYork.",
    "lang_original": "The green Toyota is west of Empire Apartments and north of Statesman Apartments and the red Honda is south of Statesman Apartments and west of The York.",
    "map_link": "https://drive.google.com/uc?export=view&id=1NHB3DiYUZtpIdgLnftnr8fU8Ur7Pk0Tm",
    "map_name": "washington_dc",
    "map_num": "2",
    "obj1_name": "rcar",
    "obj2_name": "gcar",
    "rcar_loc": "1254",
    "relations": [
        [
            "GreenToyota",
            "StatesmanApartments",
            "is with with with north of"
        ],
        [
            "RedHonda",
            "TheYork",
            "is with with with west of"
        ],
        [
            "GreenToyota",
            "EmpireApartments",
            "is with with west of"
        ],
        [
            "RedHonda",
            "StatesmanApartments",
            "is with south of"
        ]
    ]
}

Here, the relations field corresponds to the set of (figure, relation, ground) tuples. lang_original is the language input by AMT. lang shows an intermediate step in the process of obtaining the relations, where the target object names and landmark names are automatically replaced by their corresponding symbols, using dependency parsing and noun phrase identification by spaCy. The other fields are self-explanatory.

sg_parsed_annotated

We randomly selected 20 language descriptions per city, and manually annotated their spatial relations, so as to run object search experiments without the factor of parsing, since our method is not specific to our parsing pipeline.

Each spatial language description has a sg_{##}.json file, with the same format as described above. Except that the tuples in relations are annotated manually.

frame_of_ref

Contains front-positive.pkl and left-positive.pkl under blind/. Each contains a list of dictionaries of the format as in the following example; Each corresponds to a data sample with a frame of reference annotation.

 {'map_name': 'washington_dc',
  'obj_in_rel': 'bike',
  'obj_loc': (1, 30),
  'landmark_symbol': 'TheYork',
  'spatial_keyword': 'behind',
  'frame_of_refs': [[13, 30], 0.03795644518831435]}
  'landmarks_mentioned': {'20thStreetNorthwest',
                          'StatesmanApartments',
                          'TheYork'},
  'hint': 'The bike is in the Statesman apartments, directly behind The York if you are looking from 20th Street Northwest.The green car is in the VIP section behind the United Church if you are looking from 20th street Northwest.',
  'relations': [['RedBike', '20thStreetNorthwest', 'is looking from'],
   ['GreenToyota', '20thStreetNorthwest', 'is looking from'],
   ['RedBike', 'TheYork', 'is behind'],
   ['RedBike', 'StatesmanApartments', 'is in']],1
  'obj1_name': 'bike',
  'obj1_loc': '1210',
  'obj2_name': 'gcar',
  'obj2_loc': '154'}

This is essentially the same as the sg_{##}.json files, except there are a couple of new fields, notably spatial_keyword, frame_of_refs and landmarks_mentioned.

front-positive.pkl contains data samples where spatial_keyword is either "front" or "behind". left-positive.pkl contains data samples where spatial_keyword is either "left" or "right".

Note that the data for a preposition can be loaded using SpatialRelationDataset

preposition = "front"
maps = ["cleveland", "denver", "austin", "honolulu", "washington_dc"]
dataset = SpatialRelationDataset.build(preposition, maps, datadir, fields=fields)
for i in range(len(dataset)):
    sample = dataset[i]
    ... # sample is a dictionary in the fomat above.

mapinfo

Contains various data about each city map. For each city (e.g. austin), these data files are under mapinfo/{city_name}:

• {city_name}_diam_100m.PNG: Map screenshot. Covers an area of roughly 200m x 200m

• cardinal_to_limit.json: Maps map name to the longitude/lattitude of the four corners of the map screenshot {map_name -> {"N": (0.02200,0.400) ...}}

• idx_to_cell.json: Maps location index (the integer in, for example, rcar_loc) to the longitude lattiude coordinates of the four corners of the cell corresponding to that index. idx -> {"nw": [41.82, -71.41]}, ...

• name_to_idx.json: Maps landmark name to the map location indices occupied by the landmark.

• name_to_symbols.json: Maps the landmark name to a symbol string. For example "Emergency Dentist Austin" to "EmergencyDentistAustin".

• symbol_to_name.json: The reverse of name_to_symbol.json.

• pomdp_to_idx.json: Maps a grid cell coordinate to a location index. For example (3,4) to 163; So called "POMDP" because it's the underlying representation of possible locations on the map in our POMDP formulation).

• streets.json: Contains a list of landmark symbols that are considered streets.

• symbol_to_synonym.json: Defines a mapping from landmark symbol to several potential word phrases that refer to this landmark symbol, that are not necessarily the exact full name of the landmark.

• excluded_symbols.json: Maps map_name to symbols excluded.

To save your time parsing all of these data, we provide MapInfoDataset that conveniently loads this data and provides a straightforward interface. For example:

mapinfo = MapInfoDataset()
mapinfo.load_by_name("austin")
landmark_locs = mapinfo.landmark_footprint("LavacaPlaza", "austin")

sg_parsed_annotated_seed_matching

This is identical data to sg_parsed_annotated. This is only included because that the seeds (i.e. the integer at the end of the JSON file for each spatial graph) correspond to the trials in the experiment results.