Parser Recorder file

[marcusvinicius178]

Hi My System information is below:

• OS: Ubuntu 18.04
• Apollo installed from binary. I mean I am using Docker. Installed using this tutorial to clone the repo, etc: https://www.svlsimulator.com/docs/archive/2020.06/apollo-master-instructions/
• Apollo version: Apollo.Auto 6.0

Steps to reproduce the issue:

Hi I am trying to plot data from a simulation. I need to compare different planners. For that I need to take for example:

• Odometry data
• GNSS data
• Control Data
• Planning Data
• etc

At first I have recorded the channels using the cyber_record tool, through the command below:

cyber_recorder record -all

Or just triggering the record tab on Dreamview panel. Then I got all the recorded channels data stored on this folder:

Afterwards I followed this tutorial https://github.com/ApolloAuto/apollo/tree/master/modules/tools/record_parse_save
to get info from the recorded data and finally parse this data to .txt format. My idea is later get this .txt data convert to .csv or DataFrames to plot the data. However unfortunately I got the error: NO SUCH FILE OR DIRECTORY below after following the tutorial commands:

The record file was stored successfully and the info from the messages recorded from multiple channels can also be vizualized above. However the record_parse_save.py cannot find the parser_params.yaml file, which exist and is located in the correct folder, as you can check below:

I do not know what is wrong and why the module record_parse_save.py cannot open the folder and find the parser_params.yaml file.

Is this code wrong? Can someone from Apollo team check this for me, what should I do to get this module working and take the parsed data that I need?

Thanks in advance

[daohu527]

1. You should modify the input path in func read_parameters. The latest code has fixed this bug!
   

   apollo/modules/tools/record_parse_save/record_parse_save.py

   Line 134 in a75f30b

   parse_dict = read_parameters('modules/tools/record_parse_save/parser_params.yaml')

2. Also if you want to parse lidar you should modify the config in parser_params.yaml, seems can't parse all message type once.

[marcusvinicius178]

Hi @daohu527 thanks very much for the answer.

1. You should modify the input path in func read_parameters. The latest code has fixed this bug!

You were right, part of this error is fixed! I imagine when I install Apollo 6.0 (latest release) following this tutorial: https://www.svlsimulator.com/docs/archive/2020.06/apollo-master-instructions/
It gets an old release.

Now the MOST IMPORTANT! , I have replaced the record_parse_save.py I had for that one you sent me on the provided link above, then I have runned again the command to parse the files, and I got this error: No module named 'modules drivers'

Well I believe this parser works for a real test, right? I should have the pyshical camera, radar or lidar? Unfortunately I am working just with the SVL simulator data. I do not have these sensors, I am just executing simulations without a real car.

Or is this error regarding other issue? Maybe this old release repository which I have dowloaded ?

2. Also if you want to parse lidar you should modify the config in parser_params.yaml, seems can't parse all message type once.

No I don't wish lidar, camera or radar data. I wish to parse other data recorded, such as Odometer, IMU, control, planning, time. I

I tried to modify the parser_params.yaml to accept IMU data, as you can check below:

But unfortunately Apollo Team did not create an parse_imu.py module, just the code for camera, lidar and radar:

Could you provide me the other sensors/channels parsers please?
Honestly I have no idea how to properly create these parsers for the different channels: odometry, imu, control, planning, etc.

To complete my master thesis I must to compare 2 different path planners and I need to plot the channels data I described to compare graphically against each other. I wish to plot the data from these channels:

Considering the listener option and plot functions available on Apollo Dreamview are Dynamic and online plot, I believe the only way to plot these data would be:

1- Record the Channels using cyber recorder
2- Use the parser tool to get this Channel record data parsed with .txt format
3- Convert it to CSV or Data frames using python, to plot them more easily.

However as I said I do not know to create a parser for the IMU, Odometry, Control, Planning and the other channels....

Is it another feasible/possible way to plot this data for further presentation on my master Thesis?

I would like to get those plots available on PNC monitor in Dreamview. They are pretty good and would be enough for me!!! How is it possible?

Thanks in advance

[daohu527]

1. No, it use for parse the recording package.
2. If you want to parse other message, you should write new code.

[marcusvinicius178]

@daohu527 thanks again for your attention!

I understand but the recorded package contains the other channels Data (Odometer, IMU, Planning, Control, etc). When the parser tool parses a recorded file for the Radar, lidar or camera data do These other data channels come parsed together in .txt format?

I could not check the parser tool functionality as I have displayed you. I had an error NO MODULE NAMED MODULE DRIVERS when I executed the parser command.

1 - Why do I get this error and how to fix it?
Again: I don't have data from real sensor (camera, lidar or radar) but the data is coming from LG SVL simulator. Is this the error related to this or not? Other error in Code?

I would like to see a parsed recorded data...how it comes, maybe the other channels Data I wish come along with it.

2- Finally if you can really assure me that the other channels Data ( Odometer, IMU, etc) do not come after parse the .record file for radar for example... How can get these other channels Data (Odometer, IMU, control, planning, etc) to plot it ? I mean it must exist some way, instead generates parser_channels.py modules code from scratch such as radar, camera and lidar. Because similar Data is plotted on PNC monitor in Dreamview. I just wish to plot similar graphs as those ones ....

I ask this because when I used the cyber recorder, I have recorded all channels, therefore all channels Data must be inside this .record file
Then I am imagining for example that the parser tool would extract a .txt with several columns, for example:

IMU:1.20, Odometer: 0.2, Velocity:20, Control:1.4
IMU:1.34, Odometer:4, Velocity:35, control:0.2

and so on. That is why I wish to parse some .record file to check if these data comes along radar, camera or lidar data, however the Driver Module error does not let me to complete this task ...

3- Do you have a .txt parsed record file example to share with me? I would like to take a look

Thanks in advance on questions 1,2,3

[marcusvinicius178]

@daohu527 a last addition of the previous question. As I am not sure if the Parser tool will be enough to get the channels data, I took a look into the plot tools from Apollo: https://github.com/ApolloAuto/apollo/blob/master/modules/tools/realtime_plot/realtime_plot.py

A - realtime_plot.py : This module is the responsible to plot the graphics available on PNC monitor on Dreaview, corrcet?
It subscribes chassis, ADCTrajectory and other data. As far as I understood it get the car speed, steer angle, curvature and updates this data, and plot this data dynamically syncronized with the running time on the simulation side. It contains the callback functions required to get this data and the matplotlib is responsible to create the axis, labels, the plot design...while the cyber monitor is not shutdow,, in other words, a dynamic plot.

I have also checked the modules:

B) plot_trace: plot_planning_result.py and plot_trace.py: Plot the car's localization: https://github.com/ApolloAuto/apollo/tree/master/modules/tools/plot_trace
C) plot_control: plot_control.py and realtime_test.py : https://github.com/ApolloAuto/apollo/tree/master/modules/tools/plot_control

D) plot_planning: https://github.com/ApolloAuto/apollo/tree/master/modules/tools/plot_planning

They are composed of numerous valuable modules, that subscribe the channels data and plot these channels dynamically:

I wish to know how to plot this data for specific window/amount of time. Let me say I wish to plot from:

The beginning of the simulation: Time = 0
End of the simulation: Time = 45 Seconds

I wish to plot then for example data for the imu channel, that as we can check in figure above is defined on imu_speed.py for example. But I wish a STATIC Graphic from all the data, I wish to have a LIST of all these IMU data in each second (TimeStamp), therefore I would have 45 points of IMU data to plot against the Time (45 seconds) or against other parameter (acc_jerk for example).

At the end I wish plot a graphic like this:

I would like similar graphics for parameters, channels, such as imu, speed, planning, jerk, steering, time, etc

I need to put this static graphic to put on the thesis, therefore therealtime plot does not fit my needsm unfortunately.

Which Snippets of code from these modules must I change to make these modules plot a "static" graphics from the data?
Are there already implemented examples available?
Could you please give a direction to plot all this "static" graphics?
I am very grateful for the material you could provide me.

Thanks in advance

[daohu527]

there is a cyber record parser tool for now.

Closed due to inactivity. If the problem persists, pls feel free to reopen it or create a new one and refer to it.