question on (example) usage

[OlgerSiebinga]

According to the submitted paper, with sbp-env "one can quickly swap out different components to test novel ideas" and "validate ... hypothesis rapidly". However, from the examples in the documentation, it is unclear to me how I can obtain performance metrics on the planners when a run a test.

Is there a way to save such metrics to a file or print them when running planners in sbp-env? If not, this might be a nice feature to implement in a future version. Otherwise, you could consider adding an example to the documentation on how to compare different planners in the same scenario.

(this question is part of the JOSS review openjournals/joss-reviews#3782)

[soraxas]

Thanks for the suggestion! Initially the stats are being displayed in the tqdm bar. The PR added new options to stream all results to a csv. You can now run

python main.py birrt maps/room1.png --save-output

and by default it will save a timestamped csv under runs/ folder (output file is customisable via --output-dir=STATS_DIR)

[soraxas]

Example output:

"nodes","time","cc_feasibility","cc_visibility","invalid_feasibility","invalid_visibility","c_max"
"1","0.006814479827880859","1","3","0","0","inf"
"1","0.007756233215332031","2","4","0","1","inf"
"1","0.008636951446533203","3","5","0","2","inf"
"2","0.009899616241455078","5","7","1","2","inf"
"2","0.011170148849487305","6","8","1","3","inf"
"2","0.011910200119018555","7","9","1","4","inf"
"2","0.01260828971862793","9","10","2","5","inf"
"3","0.013478755950927734","11","12","3","5","inf"
"3","0.014781475067138672","13","13","4","6","inf"
"4","0.016809701919555664","14","15","4","6","inf"
"4","0.01788020133972168","15","16","4","7","inf"
"5","0.018869400024414062","16","18","4","7","inf"
"6","0.02054286003112793","17","25","4","7","20.0"
"7","0.021434545516967773","19","27","5","7","20.0"
"8","0.022353410720825195","20","30","5","7","20.0"
"9","0.02368450164794922","22","32","6","7","20.0"
"10","0.02534031867980957","24","34","7","7","20.0"
"11","0.026474714279174805","26","36","8","7","20.0"
"12","0.028153419494628906","29","38","10","7","20.0"
"13","0.029124975204467773","30","40","10","7","20.0"
"14","0.03206920623779297","31","42","10","7","20.0"
...

where it saves a line per iteration.

[OlgerSiebinga]

Great, this looks like a very nice solution. However, it does not seem to work as intended on my side and I'm not sure why. When I run the following command:

python main.py rrt maps\room1.png start 100,100 goal 350,350 --save-output --output-dir=stats

The result is an empty log file in a new folder called runs. So there are two issues, the --output-dir parameter seems to be ignored and the file is empty. The folder name could be easily fixed by sending it to the function get_non_existing_filename. But I cannot figure out why the written log file is empty. (A minor other issue is that the file has no extension, adding .csv would be good)

It might be relevant that I'm on Windows 10, if it works on Linux I can try to investigate why it does not work on Windows.

Finally, I was wondering if you are planning to add this functionality to the quick start part of the documentation. The documentation is complete as is, but an example of how to use this functionality would be very valuable in my opinion.

Let me know if I can help troubleshooting by trying out specific things/fixes.

[soraxas]

Thanks @OlgerSiebinga for your thorough details on the matter! I have found out that the empty logs in windows is due to the use of colon : character. It looks like the stats.csv works when I replace that invalid character. Documentation has also been added!

[OlgerSiebinga]

Great! This fixed the issue! The example in the documentation is also very helpful.