Unable to load the semantic information from HM3D dataset

[YYDreamzure]

Habitat-Sim version

[v0.2.5]

I have been trying to load the HM3D scenes into habitat-sim. The dataset is fully downloaded including the scene .glb, semantic .glb and .txt, and configuration file.

However, the semantics of the scene doesnt seem to function properly:
Referring to scene: oEPjPNSPmzL

def make_cfg(sim_settings):

scene_graph = habitat_sim.SceneGraph()
sim_cfg = habitat_sim.SimulatorConfiguration()
sim_cfg.gpu_device_id = 0
sim_cfg.scene_id = sim_settings["scene"]
sim_cfg.scene_dataset_config_file = sim_settings["scene_dataset"]
sim_cfg.enable_physics = sim_settings["enable_physics"]
sim_cfg.load_semantic_mesh = sim_settings["semantic_mesh"] #set to True

sensor_specs = []

color_sensor_spec = habitat_sim.CameraSensorSpec()
color_sensor_spec.uuid = "color_sensor"
color_sensor_spec.sensor_type = habitat_sim.SensorType.COLOR
color_sensor_spec.resolution = [sim_settings["height"], sim_settings["width"]]
color_sensor_spec.position = [0.0, sim_settings["sensor_height"], 0.0]

color_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
color_sensor_spec.orientation = sim_settings["orientation"]
sensor_specs.append(color_sensor_spec)

semantic_sensor_spec = habitat_sim.CameraSensorSpec()
semantic_sensor_spec.uuid = "semantic_sensor"
semantic_sensor_spec.sensor_type = habitat_sim.SensorType.SEMANTIC
semantic_sensor_spec.resolution = [sim_settings["height"], sim_settings["width"]]
semantic_sensor_spec.position = sim_settings["sensor_height"] * habitat_sim.geo.UP
semantic_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
semantic_sensor_spec.orientation = sim_settings["orientation"]
sensor_specs.append(semantic_sensor_spec)

agent_cfg = habitat_sim.agent.AgentConfiguration()
agent_cfg.sensor_specifications = sensor_specs
agent = habitat_sim.Agent(scene_graph.get_root_node().create_child(), agent_cfg)

return habitat_sim.Configuration(sim_cfg, [agent_cfg])

CODE:
scene = sim.semantic_scene
print(f"House has {len(scene.levels)} levels, {len(scene.regions)} regions and {len(scene.objects)} objects")
print(f"House center:{scene.aabb.center} dims:{scene.aabb.sizes}")

OUTPUT:
The NavMesh bounds are: (array([-11.723581 , -0.18547955, -2.3691924 ], dtype=float32), array([4.749757 , 4.3199515, 4.6835546], dtype=float32))
House has 0 levels, 10 regions and 301 objects
House center:[0. 0. 0.] dims:[-inf -inf -inf]

And when I run simulations, I get this error:
[16:20:15:076408]:[Assets] ResourceManager.cpp(2941)::joinSemanticHierarchy : Could not get the GenericSemanticMeshData

I did put all the files in the same folder. May I know where might have gone wrong?

❓ Questions and Help

[janblumenkamp]

I have a similar problem. It is not clear to me how the height of an agent relative to the floor can be determined, particularly in scenes with multiple floors.

[aclegg3]

@YYDreamzure levels are not annotated in HM3D, regions are in some cases annotated, but not matched to a semantic class. In the paper we inferred region labels from furniture semantics, but did not propogate these inferences into the dataset since they are estimates rather than ground truth.

@janblumenkamp The height of an agent relative to the floor is fixed. This is part of the sensor config.

color_sensor_spec.position = [0.0, sim_settings["sensor_height"], 0.0]

The agent's position is actually the position of it's feet in contact with the ground via navmesh approximation. There are some techniques to estimate the levels and you can use the bounds of the navmesh to determine the lowest and highest ground Y values in the scene if you need that information.

[janblumenkamp]

Thanks for clarifying!
I found that using get_random_navigatable_point gives a good indication of floors. I hacked this function to return randomly sampled points on different floors:

def sample_random_points(sim, volume_sample_fac=1.0, significance_threshold=0.2):
    scene_bb = sim.get_active_scene_graph().get_root_node().cumulative_bb
    scene_volume = scene_bb.size().product()
    points = np.array([sim.pathfinder.get_random_navigable_point() for _ in range(int(scene_volume * volume_sample_fac))])

    hist, bin_edges = np.histogram(points[:, 1], bins='auto')
    significant_bins = (hist / len(points)) > significance_threshold
    l_bin_edges = bin_edges[:-1][significant_bins]
    r_bin_edges = bin_edges[1:][significant_bins]
    points_floors = {}
    for l_edge, r_edge in zip(l_bin_edges, r_bin_edges):
        points_floor = points[(points[:, 1] >= l_edge) & (points[:, 1] <= r_edge)]
        height = points_floor[:, 1].mean()
        points_floors[height] = points_floor
    return points_floors

It's sampling a large number of points in the scene, and then clusters them along the height axis. Sufficiently large clusters are points on the same height. This seems to work reasonably well across all scenes.

[aclegg3]

Thanks for sharing, @janblumenkamp. We've done similar things also when needing a floor heuristic. 👍

[aclegg3]

FYI, some notes on annotating regions that are relevant to this discussion: #2364 (comment)

Note these examples are intended to help you annotate your own scenes, not to load some existing data. HM3D does not currently provide floor and region annotations.