Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nuboard not displaying any information #67

Closed
kensukenk opened this issue Jun 27, 2022 · 26 comments
Closed

Nuboard not displaying any information #67

kensukenk opened this issue Jun 27, 2022 · 26 comments
Assignees

Comments

@kensukenk
Copy link

Hi! When I run the nuplan_framework.ipynb notebook, I can get all the models to train but not shows up when I launch Nuboard. I get some messages when I train and simulate that maybe suggest the error?

"The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work.
    Check out the dashboard_agent.log to see the detailed failure messages."
    dashboard_agent.log

This is what appears when I try to launch Nuboard. It seems that there are two experiments, but no data shows up at all.
Screenshot from 2022-06-27 11-37-48

I know that Nuboard is under development, but I was wondering if I was supposed to be able to see anything at all? If not, where would I be able to find the raw data from simulation and training?

@patk-motional
Copy link
Collaborator

Hi @kensukenk,

Yes, you should be able to see something. I will ask my colleague @kokseang-motional to take a look. In the meantime, can you share the command you used to

  1. Run the simulation
  2. Run nuBoard

@kokseang-motional
Copy link
Contributor

kokseang-motional commented Jun 28, 2022

Hey @kensukenk
Would u mind sharing the commands u run simulation and nuboard?
Also, did u see any error messages in cli? Did overview and scenario tabs output sth?
Thanks for reporting this

@kensukenk
Copy link
Author

I just ran jupyter notebook ~/nuplan-devkit/tutorials/nuplan_framework.ipynb in my terminal and then ran every cell down the notebook. Nothing outputs in any tab in nuboard. This is my version of the notebook in case I accidentally changed anything but I don't believe I did:
nuplan_framework.zip

I just installed nuplan yesterday, so to be clear, in my home directory I have both ~/nuplan-devkit and ~/nuplan?
Screenshot from 2022-06-27 21-21-25
Screenshot from 2022-06-27 21-21-14

@kokseang-motional
Copy link
Contributor

Hi @kensukenk
Can u try changing this line
f'simulation_path={[nuboard_file_1, nuboard_file_2]}'
to
f'simulation_path={[nuboard_file_1]}' only?

@kensukenk
Copy link
Author

I still have the same issue only using nuboard_file_1. In case this is useful, here are the aggregated metrics for the simulation simple experiment.
metrics.csv

@kokseang-motional
Copy link
Contributor

Hi @kensukenk
What happened if u run simulation in cli and then nuboard? Does it show results correctly?

@kensukenk
Copy link
Author

It still does not work. Does it matter that I get certain runtime warnings such as "RuntimeWarning: invalid value encountered in subtract
(wrapped_fn pid=887202) diff_b_a = subtract(b, a)" or "(raylet) E0704 12:01:28.406616966 887721 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers"?

@patk-motional
Copy link
Collaborator

@kensukenk can you post your full stack trace?

@kensukenk
Copy link
Author

This is the stack trace for simulation:

python nuplan/planning/script/run_simulation.py \

+simulation=open_loop_boxes \
planner=simple_planner \
scenario_builder=nuplan_mini \
scenario_filter=all_scenarios \
scenario_filter.scenario_types="[near_multiple_vehicles, on_pickup_dropoff, starting_unprotected_cross_turn, high_magnitude_jerk]" \
scenario_filter.num_scenarios_per_type=10 \

/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
warnings.warn(
Global seed set to 0
INFO:nuplan.planning.script.builders.main_callback_builder:Building MultiMainCallback...
INFO:nuplan.planning.script.builders.main_callback_builder:Building MultiMainCallback: 3...DONE!
2022-07-04 12:01:04,158 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool...
2022-07-04 12:01:05,487 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:72} Starting ray local!
2022-07-04 12:01:07,888 INFO services.py:1374 -- View the Ray dashboard at http://10.23.245.54:8265
2022-07-04 12:01:10,061 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:100} Worker: RayDistributed
2022-07-04 12:01:10,061 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Number of nodes: 1
Number of CPUs per node: 16
Number of GPUs per node: 1
Number of threads across all nodes: 16
2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE!
2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:29} Building experiment folders...
2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:32}

Folder where all results are stored: /home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

2022-07-04 12:01:10,063 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:58} Building experiment folders...DONE!
2022-07-04 12:01:10,063 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder...
(raylet) E0704 12:01:12.877570964 887631 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
2022-07-04 12:01:12,998 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:166} Loaded maps DB
2022-07-04 12:01:15,751 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:176} Loaded 72 log DBs
2022-07-04 12:01:15,765 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE!
2022-07-04 12:01:15,766 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_callback_builder.py:24} Building AbstractCallback...
2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_callback_builder.py:36} Building AbstractCallback: 2...DONE!
2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:45} Building simulations...
2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:19} Building ScenarioFilter...
2022-07-04 12:01:15,775 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:22} Building ScenarioFilter...DONE!
2022-07-04 12:01:15,775 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:52} Extracting scenarios...
(raylet) E0704 12:01:18.203487316 887706 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
2022-07-04 12:01:18,310 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:54} Extracting scenarios 34...DONE!
2022-07-04 12:01:18,319 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:58} Building metric engines...
2022-07-04 12:01:18,800 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:60} Building metric engines...DONE
2022-07-04 12:01:18,807 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:64} Building simulations from 34 scenarios...
2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:112} Building simulations...DONE!
2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_simulation.py:75} Running simulation...
2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:128} Executing runners...
2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:83} Starting 34 simulations using RayDistributed!
Ray objects: 0%| | 0/34 00:00<?, ?it/s /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887197) warnings.warn(
(pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887207) warnings.warn(
(pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887195) warnings.warn(
(pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887206) warnings.warn(
(pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887200) warnings.warn(
(pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887196) warnings.warn(
(pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887201) warnings.warn(
(pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887190) warnings.warn(
(pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887198) warnings.warn(
(pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887199) warnings.warn(
(pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887204) warnings.warn(
(pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887202) warnings.warn(
(pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887193) warnings.warn(
(pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887191) warnings.warn(
(pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887192) warnings.warn(
(pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=887194) warnings.warn(
(raylet) E0704 12:01:28.406616966 887721 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
(raylet) E0704 12:01:47.294714735 887731 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
(wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
(wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
(wrapped_fn pid=887201) diff_b_a = subtract(b, a)
(wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
(wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
(wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
Ray objects: 3%|▉ | 1/34 [00:57<31:48, 57.84s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
(wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
(wrapped_fn pid=887202) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887202) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887202) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887202) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
(wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
(wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
(raylet) E0704 12:02:22.522624360 887781 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
2022-07-04 12:02:23,239 WARNING worker.py:1257 -- (ip=10.23.245.54) The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work.
    Check out the dashboard_agent.log to see the detailed failure messages.
    (wrapped_fn pid=887204) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887204) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887204) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887204) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887202) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887207) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887207) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887207) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887207) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887190) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887190) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887190) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887190) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887194) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887194) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887194) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887194) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887199) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887199) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887199) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887199) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887206) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887206) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887206) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887206) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887198) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887198) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887198) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887198) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887191) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887191) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887191) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887191) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887197) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887197) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887197) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887197) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887200) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887200) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887200) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887200) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887202) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 6%|█▊ | 2/34 [01:13<17:43, 33.23s/it](wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887193) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 9%|██▋ | 3/34 [01:15<09:36, 18.59s/it](wrapped_fn pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887204) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 12%|███▌ | 4/34 [01:22<07:01, 14.03s/it](wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887199) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 15%|████▍ | 5/34 [01:23<04:32, 9.41s/it](wrapped_fn pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887198) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 18%|█████▎ | 6/34 [01:23<02:56, 6.31s/it](wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887196) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887196) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887196) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887196) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887206) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 21%|██████▏ | 7/34 [01:26<02:18, 5.13s/it](wrapped_fn pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887190) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 24%|███████ | 8/34 [01:27<01:41, 3.91s/it](wrapped_fn pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887200) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    (wrapped_fn pid=887197) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887197) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887194) diff_b_a = subtract(b, a)
    Ray objects: 26%|███████▉ | 9/34 [01:28<01:16, 3.07s/it](wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 29%|████████▌ | 10/34 [01:28<00:51, 2.16s/it](wrapped_fn pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887191) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 35%|██████████▏ | 12/34 [01:29<00:30, 1.40s/it](wrapped_fn pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887207) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 38%|███████████ | 13/34 [01:30<00:25, 1.22s/it](wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887192) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887192) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887192) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887192) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887196) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 41%|███████████▉ | 14/34 [01:47<01:46, 5.34s/it](wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887195) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887195) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887195) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887195) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887192) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 44%|████████████▊ | 15/34 [02:19<03:58, 12.57s/it](wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887199) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric
    (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887195) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 47%|█████████████▋ | 16/34 [02:43<04:45, 15.85s/it](wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887199) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 50%|██████████████▌ | 17/34 [02:46<03:24, 12.03s/it](wrapped_fn pid=887194) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric
    (wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887193) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887204) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 53%|███████████████▎ | 18/34 [02:48<02:26, 9.17s/it](wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 56%|████████████████▏ | 19/34 [02:48<01:37, 6.53s/it](wrapped_fn pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887194) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 59%|█████████████████ | 20/34 [02:51<01:18, 5.62s/it](wrapped_fn pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887200) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 62%|█████████████████▉ | 21/34 [02:52<00:52, 4.03s/it](wrapped_fn pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887202) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 65%|██████████████████▊ | 22/34 [02:52<00:36, 3.07s/it](wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887207) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 68%|███████████████████▌ | 23/34 [02:56<00:36, 3.36s/it](wrapped_fn pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887206) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 71%|████████████████████▍ | 24/34 [03:01<00:36, 3.66s/it](wrapped_fn pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887190) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 74%|█████████████████████▎ | 25/34 [03:02<00:25, 2.88s/it](wrapped_fn pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887198) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 76%|██████████████████████▏ | 26/34 [03:02<00:16, 2.07s/it](wrapped_fn pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887201) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 79%|███████████████████████ | 27/34 [03:03<00:11, 1.63s/it](wrapped_fn pid=887197) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887197) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 82%|███████████████████████▉ | 28/34 [03:05<00:11, 1.93s/it](wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887191) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 85%|████████████████████████▋ | 29/34 [03:09<00:11, 2.40s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
    (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna(
    (wrapped_fn pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887196) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 88%|█████████████████████████▌ | 30/34 [03:19<00:19, 4.84s/it](wrapped_fn pid=887192) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric
    (wrapped_fn pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887192) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 91%|██████████████████████████▍ | 31/34 [03:22<00:12, 4.08s/it](wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887193) diff_b_a = subtract(b, a)
    Ray objects: 94%|███████████████████████████▎ | 32/34 [03:25<00:07, 3.89s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887195) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 97%|████████████████████████████▏| 33/34 [03:30<00:04, 4.08s/it](wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation...
    (wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract
    (wrapped_fn pid=887199) diff_b_a = subtract(b, a)
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics!
    (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics!
    Ray objects: 100%|█████████████████████████████| 34/34 [03:38<00:00, 6.42s/it]
    2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:100} Number of successful simulations: 34
    2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:101} Number of failed simulations: 0
    2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:136} Finished executing runners!
    2022-07-04 12:04:58,418 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:64} Saved simulation reports to /home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03/simulations_report.parquet
    2022-07-04 12:04:58,418 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/time_callback.py:27} Simulation duration: 00:03:54 [HH:MM:SS]
    2022-07-04 12:04:59,230 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_file_callback.py:55} Metric files integration: 00:00:00 [HH:MM:SS]
    2022-07-04 12:04:59,355 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:45} Running metric aggregator: weighted_average_metric_aggregator
    2022-07-04 12:04:59,416 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:53} Metric aggregator: 00:00:00 [HH:MM:SS]
    2022-07-04 12:04:59,417 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_simulation.py:77} Finished running simulation!

@kensukenk
Copy link
Author

training:

python nuplan/planning/script/run_training.py experiment_name=raster_experiment py_func=train +training=training_raster_model scenario_builder=nuplan_mini scenario_filter.limit_total_scenarios=500 lightning.trainer.params.max_epochs=10 data_loader.params.batch_size=8 data_loader.params.num_workers=8
/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
warnings.warn(
Global seed set to 0
2022-07-04 11:56:03,494 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:17} Building experiment folders...
2022-07-04 11:56:03,495 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:19} Experimental folder: /home/kensuke/nuplan/exp/exp/raster_experiment/2022.07.04.11.56.03
2022-07-04 11:56:03,495 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool...
2022-07-04 11:56:03,497 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:72} Starting ray local!
2022-07-04 11:56:05,882 INFO services.py:1374 -- View the Ray dashboard at http://10.23.245.54:8265
2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:100} Worker: RayDistributed
2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Number of nodes: 1
Number of CPUs per node: 16
Number of GPUs per node: 1
Number of threads across all nodes: 16
2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE!
2022-07-04 11:56:08,033 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/experiments/training.py:42} Building training engine...
2022-07-04 11:56:08,033 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:18} Building TorchModuleWrapper...
2022-07-04 11:56:08,609 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/timm/models/helpers.py:188} Loading pretrained weights from url (https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-rsb-weights/resnet50_a1_0-14fe96d1.pth)
2022-07-04 11:56:08,707 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/timm/models/helpers.py:208} Converted input conv conv1 pretrained weights from 3 to 4 channel(s)
2022-07-04 11:56:08,745 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:21} Building TorchModuleWrapper...DONE!
2022-07-04 11:56:08,745 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:18} Building Splitter...
2022-07-04 11:56:09,957 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:21} Building Splitter...DONE!
2022-07-04 11:56:09,957 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder...
(raylet) E0704 11:56:10.899484671 884432 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
2022-07-04 11:56:12,854 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:166} Loaded maps DB
2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:176} Loaded 72 log DBs
2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE!
2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:19} Building ScenarioFilter...
2022-07-04 11:56:15,852 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:22} Building ScenarioFilter...DONE!
Ray objects: 0%| | 0/16 00:00<?, ?it/s E0704 11:56:16.699473880 884547 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
(pid=883990) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883990) warnings.warn(
(pid=884003) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=884003) warnings.warn(
(pid=884000) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=884000) warnings.warn(
(pid=883999) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883999) warnings.warn(
(pid=883993) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883993) warnings.warn(
(pid=883989) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883989) warnings.warn(
(pid=884005) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=884005) warnings.warn(
(pid=883997) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883997) warnings.warn(
(pid=883991) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883991) warnings.warn(
(pid=884001) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=884001) warnings.warn(
(pid=883995) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883995) warnings.warn(
(pid=883996) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883996) warnings.warn(
(pid=883992) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883992) warnings.warn(
(pid=884004) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=884004) warnings.warn(
(pid=883998) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883998) warnings.warn(
(pid=883994) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
(pid=883994) warnings.warn(
(raylet) E0704 11:56:27.614408888 884559 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
Ray objects: 50%|███████████████ | 8/16 00:29<00:10, 1.34s/it E0704 11:56:45.648005160 884581 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
Ray objects: 100%|█████████████████████████████| 16/16 [00:34<00:00, 2.17s/it]
2022-07-04 11:56:51,084 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_builder.py:151} Extracted 500 scenarios for training
2022-07-04 11:56:51,088 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:19} Building callbacks...
2022-07-04 11:56:51,106 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:28} Building callbacks...DONE!
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Using native 16bit precision.
2022-07-04 11:56:51,109 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_training.py:60} Starting training...
Global seed set to 0
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
2022-07-04 11:56:51,114 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:194} Added key: store_based_barrier_key:1 to store for rank: 0
2022-07-04 11:56:51,115 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:224} Rank 0: Completed store-based barrier for 1 nodes.

distributed_backend=nccl
All DDP processes registered. Starting ddp with 1 processes

2022-07-04 11:56:56,302 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/data_loader/datamodule.py:42} Number of samples in train set: 368
2022-07-04 11:56:56,314 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/data_loader/datamodule.py:42} Number of samples in validation set: 57
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

| Name | Type | Params

0 | model | RasterModel | 23.6 M

23.6 M Trainable params
0 Non-trainable params
23.6 M Total params
94.340 Total estimated model params size (MB)
/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/pytorch_lightning/callbacks/lr_monitor.py:97: RuntimeWarning: You are using LearningRateMonitor callback with models that have no learning rate schedulers. Please see documentation for configure_optimizers method.
rank_zero_warn(
Epoch 0: 0%| | 0/53 [00:00<?, ?it/s]/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
2022-07-04 11:56:57,516 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:788} Reducer buckets have been rebuilt in this iteration.
Epoch 0: 87%|█████████████ | 46/53 [00:06<00:01, 6.75it/s, loss=231, v_num=]/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/pytorch_lightning/trainer/callback_hook.py:100: LightningDeprecationWarning: The signature of Callback.on_train_epoch_end has changed in v1.3. outputs parameter has been removed. Support for the old signature will be removed in v1.5
warning_cache.deprecation(
Epoch 0: 89%|█████████████▎ | 47/53 [00:08<00:01, 5.37it/s, loss=231, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
Epoch 1: 0%| | 0/53 [00:00<?, ?it/s, loss=231, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
Epoch 1: 87%|█████████████ | 46/53 [00:08<00:01, 5.41it/s, loss=234, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
(raylet) E0704 11:57:19.226724981 885067 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
2022-07-04 11:57:19,680 WARNING worker.py:1257 -- (ip=10.23.245.54) The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work.
    Check out the dashboard_agent.log to see the detailed failure messages.
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 2: 0%| | 0/53 [00:00<?, ?it/s, loss=234, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 2: 87%|█████████████ | 46/53 [00:07<00:01, 5.88it/s, loss=221, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 3: 0%| | 0/53 [00:00<?, ?it/s, loss=221, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 3: 87%|█████████████ | 46/53 [00:08<00:01, 5.61it/s, loss=199, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 4: 0%| | 0/53 [00:00<?, ?it/s, loss=199, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 4: 87%|█████████████ | 46/53 [00:08<00:01, 5.29it/s, loss=213, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 5: 0%| | 0/53 [00:00<?, ?it/s, loss=213, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 5: 87%|█████████████ | 46/53 [00:08<00:01, 5.21it/s, loss=233, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 6: 0%| | 0/53 [00:00<?, ?it/s, loss=233, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 6: 87%|█████████████ | 46/53 [00:08<00:01, 5.14it/s, loss=229, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 7: 0%| | 0/53 [00:00<?, ?it/s, loss=229, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 7: 87%|█████████████ | 46/53 [00:08<00:01, 5.33it/s, loss=241, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 8: 0%| | 0/53 [00:00<?, ?it/s, loss=241, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 8: 87%|█████████████ | 46/53 [00:08<00:01, 5.18it/s, loss=218, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 9: 0%| | 0/53 [00:00<?, ?it/s, loss=218, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 9: 87%|█████████████ | 46/53 [00:08<00:01, 5.15it/s, loss=209, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
    Epoch 9: 100%|███████████████| 53/53 [00:14<00:00, 3.68it/s, loss=209, v_num=]

@patk-motional
Copy link
Collaborator

Hi @kensukenk,

To see the results of your simulation you need to launch nuBoard.

python nuplan/planning/script/run_nuboard.py

You will then be re-directed to your browser. From there you can upload the simulation file using the GUI. Your simulation should be here:

/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

@kensukenk
Copy link
Author

kensukenk commented Jul 5, 2022

Even after uploading the simulation file no data shows up.

@kokseang-motional
Copy link
Contributor

Hey @kensukenk
Would u mind zip and upload your simulation folder here?
/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

I suspect there is no simulation data available in the folder

@kensukenk
Copy link
Author

The zipped file is too large to upload, so here is a google drive link with the zip file.

https://drive.google.com/file/d/1roBFx-T4cMZZqp4ucr7e8G5tEGy8Mg14/view?usp=sharing

@kokseang-motional
Copy link
Contributor

Hi @kensukenk
Thanks for providing the zip file, it actually worked on my end
Can u try the following command:

python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]"

@kensukenk
Copy link
Author

Hmmm, for some reason there is still no visualization loading on my end.

1 similar comment
@kensukenk
Copy link
Author

Hmmm, for some reason there is still no visualization loading on my end.

@kokseang-motional
Copy link
Contributor

kokseang-motional commented Jul 12, 2022

hi @kensukenk
Can u show your Bokeh version?
pip show bokeh

Also, any error in your cli when change the search filters in nuBoard?

@kensukenk
Copy link
Author

Name: bokeh
Version: 2.4.3
Summary: Interactive plots and applications in the browser from Python
Home-page: https://github.com/bokeh/bokeh
Author: Bokeh Team
Author-email: info@bokeh.org
License: BSD-3-Clause
Location: /home/kensuke/anaconda3/envs/nuplan/lib/python3.9/site-packages
Requires: PyYAML, Jinja2, numpy, tornado, packaging, typing-extensions, pillow
Required-by: nuplan-devkit

And I get no errors in my cli when I change search filters. Only this when I run it:
python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]"
/home/kensuke/anaconda3/envs/nuplan/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
warnings.warn(

@kokseang-motional
Copy link
Contributor

Can u try press F12 to look at your browser console, any errors after you try searching sth?

@kokseang-motional
Copy link
Contributor

Hi @kensukenk
Could u try to reinstall the devkit again? We have few updates

@kensukenk
Copy link
Author

kensukenk commented Jul 19, 2022

Ah, so now it is a little bit different. The histogram tab still doesn't tell me anything, but now the scenarios tab displays text before I choose which scenario to pick. Afterward, it loads indefinitely it seems, but my terminal says

"(nuplan2) kensuke@Blade-15:~/nuplan-devkit$ python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]"
/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
warnings.warn(
INFO:nuplan.planning.script.builders.scenario_building_builder:Building AbstractScenarioBuilder...
INFO:nuplan.database.nuplan_db.nuplandb_wrapper:Loaded maps DB
INFO:nuplan.database.nuplan_db.nuplandb_wrapper:Loaded 72 log DBs
INFO:nuplan.planning.script.builders.scenario_building_builder:Building AbstractScenarioBuilder...DONE!
INFO:nuplan.planning.nuboard.nuboard:Opening Bokeh application on http://localhost:5006/
INFO:bokeh.server.server:Starting Bokeh server version 2.4.3 (running on Tornado 6.1)
INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled)
INFO:tornado.access:200 GET / (127.0.0.1) 164.10ms
INFO:tornado.access:200 GET /resource/style.css (127.0.0.1) 2.56ms
INFO:tornado.access:200 GET /resource/spectre.min.css (127.0.0.1) 0.68ms
INFO:tornado.access:200 GET /resource/spectre-exp.min.css (127.0.0.1) 0.33ms
INFO:tornado.access:200 GET /resource/spectre-icons.min.css (127.0.0.1) 0.23ms
INFO:tornado.access:200 GET /static/js/bokeh-gl.min.js?v=e5df31fd9010eacff0aa72d315264604b5e34972ba445acea6fce98080eecf33acf2d2986126360faaa5852813cffa16f6f6f4889923318300f062497c02da4e (127.0.0.1) 0.37ms
INFO:tornado.access:200 GET /static/js/bokeh.min.js?v=3c61e952b808bb7e346ce828a565a5f23aaf7708d034fa9d0906403813355d45bb4e8d8b0b23a93f032c76831d4f0221846f28699c7f5147caa62e0d31668314 (127.0.0.1) 0.52ms
INFO:tornado.access:200 GET /static/js/bokeh-mathjax.min.js?v=176c36fdbcd8fc1019fc828101a2804081a35baf4018d7f2633cd263156b593aa73112f400112b662daa0590138b74851bc91f1f2a5fbf5416ee8c876c3e0d0c (127.0.0.1) 1.33ms
INFO:tornado.access:200 GET /static/js/bokeh-widgets.min.js?v=8a1ff6f5aa0d967f4998d275803bbb111d928fd9f605ef9e1f30cfd021df0e77224ee3d13f83edb3a942f6e4ccc569ee5dd8951a8aa6cb600602463b90c65a87 (127.0.0.1) 1.46ms
INFO:tornado.access:200 GET /static/js/bokeh-tables.min.js?v=ae2903e57cf57f52819fdf4d938c648982b51c34f73b6e653a0f3bb3c8ab44f338505931ace43eafc1636e215492e2314acf54c54baffb47813b86b4923a7fe0 (127.0.0.1) 1.58ms
INFO:tornado.access:200 GET /resource/scripts/utils.js (127.0.0.1) 0.44ms
INFO:tornado.access:200 GET /resource/motional_logo.png (127.0.0.1) 1.10ms
INFO:tornado.access:101 GET /ws (127.0.0.1) 0.60ms
INFO:bokeh.server.views.ws:WebSocket connection opened
INFO:bokeh.server.views.ws:ServerConnection created
ERROR:bokeh.server.protocol_handler:error handling message
message: Message 'PATCH-DOC' content: {'events': [{'kind': 'ModelChanged', 'model': {'id': '1021'}, 'attr': 'value', 'new': '71dfddf88e7e51e0'}], 'references': []}
error: TypeError("a bytes-like object is required, not 'list'")
Traceback (most recent call last):
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/protocol_handler.py", line 97, in handle
work = await handler(message, connection)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/session.py", line 95, in _needs_document_lock_wrapper
result = func(self, *args, **kwargs)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/session.py", line 289, in _handle_patch
message.apply_to_document(self.document, self)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/protocol/messages/patch_doc.py", line 115, in apply_to_document
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.content, setter))
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc
return f()
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/protocol/messages/patch_doc.py", line 115, in
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.content, setter))
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/document.py", line 391, in apply_json_patch
DocumentPatchedEvent.handle_json(self, event_json, references, setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/events.py", line 259, in handle_json
handler(doc, event_json, references, setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/events.py", line 467, in _handle_json
patched_obj.set_from_json(attr, value, models=references, setter=setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/has_props.py", line 384, in set_from_json
descriptor.set_from_json(self, json, models=models, setter=setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 378, in set_from_json
self._set(obj, old, value, setter=setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 559, in _set
self._trigger(obj, old, value, hint=hint, setter=setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 637, in _trigger
obj.trigger(self.name, old, value, hint, setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/model/model.py", line 567, in trigger
super().trigger(descriptor.name, old, new, hint=hint, setter=setter)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/util/callback_manager.py", line 194, in trigger
self.document.callbacks.notify_change(cast(Model, self), attr, old, new, hint, setter, invoke)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 236, in notify_change
self.trigger_on_change(event)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 373, in trigger_on_change
invoke_with_curdoc(doc, event.callback_invoker)
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc
return f()
File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/util/callback_manager.py", line 191, in invoke
callback(attr, old, new)
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 375, in _scalar_scenario_name_select_on_change
self._update_scenario_plot()
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 312, in _update_scenario_plot
self._simulation_plots = self._render_simulations()
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 783, in _render_simulations
self._simulation_figure_data = self.simulation_tile.render_simulation_tiles(
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 266, in render_simulation_tiles
self.init_simulations(figure_sizes=figure_sizes)
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 225, in init_simulations
simulation_figure = self._create_initial_figure(figure_index=figure_index, figure_sizes=figure_sizes)
File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 192, in _create_initial_figure
simulation_log = SimulationLog.load_data(simulation_file)
File "/home/kensuke/nuplan-devkit/nuplan/planning/simulation/simulation_log.py", line 78, in load_data
data = pickle.loads(data)
TypeError: a bytes-like object is required, not 'list'
/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/protocol_handler.py:101: RuntimeWarning: coroutine 'WSHandler.send_message' was never awaited
work = connection.error(message, repr(e))
"

@kokseang-motional
Copy link
Contributor

Hi @kensukenk
Have to rerun simulation since we updated the interface

@patk-motional
Copy link
Collaborator

Close due to inactivity

@927087638
Copy link

Full of various bugs.

@jessapinkman
Copy link

hey, i met some issues, when i run nuplan_planner_tutorial.ipynb for the last cell "Launch nuboard(open in new tab), it seems stucked:
image
i have waited for more than 9 minutes as show above, and the nuboard shows below:
image
image
can u tell me how to fix it? @patk-motional @kokseang-motional

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

5 participants