Filter the simulation state takes hours

[osrf-migration]

Original report (archived issue) by Martin Dlouhy (Bitbucket: robotikacz).

---

Yes, I know that you mentioned on https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/wiki/tutorials/qual that "This process can take many minutes." ... but hours? Is it really expected? The original simulation was faster (!) and the files are relatively small (original 259MB and the "filtered" after 3hours 27MB). I think that I just missed dead-line even for the west coast midnight ...

Also is it worth it? I understand that smaller the better but the "price" is maybe too high?

I am referring to command:
gz log -f ~/.gazebo/log/2018-12-22T040945.151018/gzserver/state.log --filter .pose/.pose -z 60 -o ~/.gazebo/log/2018-12-22T040945.151018/gzserver/subt_tunnel_qual_sim_state.log

[osrf-migration]

Original comment by Martin Dlouhy (Bitbucket: robotikacz).

---

• Edited issue description

[osrf-migration]

Original comment by Martin Dlouhy (Bitbucket: robotikacz).

---

OK, submitted "raw" status was gziped to 229MB and filtered version 22MB, but the conversion took 8 hours! The simulation itself was around 50 minutes.

It is really necessary to re-sample original poses to 60Hz?

[osrf-migration]

Original comment by Martin Dlouhy (Bitbucket: robotikacz).

---

p.s. simple reading original gazebo log file with Python sax, concatenating blocks, zlib.uncompress and dump sizes of each chunk took 32.520 seconds ... so if "filtering" should just drop some iterations it should be much much faster. Also I am not sure if it is necessary to dump pose of each tile for every iteration? In other words the filtered poses file could be probably much smaller.

(example how to extract readable XML is here: robotika/osgar@bbd26b5)

[osrf-migration]

Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).

---

I am working on a computer I have just bought for this. It is fairly high end
Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
4 cores, 2 threads each, turboboost up to 4Gz.

Never the less it still takes hours to do such a simple thing. There must be a bug of some kind because also loading the original log into gazebo does not work (or I haven't waited long enough).

[osrf-migration]

Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).

---

Can we use

  --record_period arg (=-1)     Recording period (seconds).
  --record_filter arg           Recording filter (supports wildcard and regular expression).

directly when launching gazebo? That would provide the log with the information that is needed for submission without the need for any filtering at the end.

[osrf-migration]

Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).

---

I tried using "-r --record_period 0.0167 --record_filter *.pose/*.pose" for the extra_gazebo_flags and the generated log file seems to have about the right size (similar to the filtered one, which btw took 362 minutes to create).

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

We are in the process of moving a new logging system, which should bypass the filtering requirement. This new logging system will be available in May. Until then, please try to work through the current constraint.

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

• changed state from "new" to "resolved"

The subt code base has transition from gazebo9 to ignition, which utilizes a different logging strategy. Filtering is no longer required, and the log files produced by simulation can be submitted as-is.

[osrf-migration]

Original comment by Alfredo Bencomo (Bitbucket: bencomo).

---

• changed state from "resolved" to "closed"