Where is the brake in the dataset? Building an E2E model...

[ashleyrudland]

Hi there - thanks so much for this dataset. I've been searching for days and this looks like the best out there for my needs. I'm currently trying to build my own E2E model from scratch using this dataset. Input: image, output: pedal, break, steering.

Hope you can help 🙏

I'm feeding my model with a DataFrame with jpeg img path (frame), steering, break and pedal. Thus I need it in this format in the DataFrame for my model to work.

If I understand correctly...

Pedal = processed_log > IMU > acceleration - correct?
Steering Angle = processed_log > CAN data > steering_angle - correct?

Question #1 Where is the break?

Also, when looking at...

Dataset_chunk_n
|
+-- route_id (dongle_id|start_time)
    |
    +-- segment_number
        |
        ...
        +-- video.hevc (video file, can be read with openpilot-tools: framereader)
        +-- processed_log/ (processed logs as numpy arrays, see format for details)

Question #2: I assume that the index in the processed_log numpy array maps to the frame in the video.hevc? e.g.

# loop through each segment, then...
fr = FrameReader(current_segment + 'video.hevc')
steering_angles = np.load(current_segment + 'processed_log/CAN/steering_angle/value')
log_index = 0

for item in steering_angles:
      steering_angle = steering_angles[log_index]
      frame_img = fr.get(frame_index, pix_fmt='rgb24')[0]
      log_index += 1

Hope that makes sense. Thanks in advance!!!

[haraschax]

Only saw this now.

You can find brake pedal in the carstate packet I think. You might have to read the raw logs.

[jdacoello]

@ashr1987 how did you find the brake pedal? Could you please describe the steps to follow? I have been trying to decode other signals of the processed_log but I do not find the way to do it. See for example issue #19

I would like for example to get the clutch, and the odometer, or any other signal available in the CAN.