How to find the ground truth flow corresponding to events?

[Hazeliii]

Hi, thanks a lot for your work.
I am confused about how to find the ground truth flow corresponding to events during training. The ground truth flows are give with timestamps, should I use the timestamps to relate them to events? And what's the usage of ' ms_to_idx' in the events.h5 file ?
Can you give me some introductions and suggestions? Thanks a lot.

[magehrig]

Hi @Hazeliii

The details about the dataset format are explained here: https://dsec.ifi.uzh.ch/data-format/
There you will find

• the explanation about ms_to_idx which you actually don't need to know if you use this python code to retrieve event data: https://github.com/uzh-rpg/DSEC/blob/main/scripts/utils/eventslicer.py
• the explanation about optical flow timestamps

Regarding timestamps in general: They are all synchronized across cameras and ground truth. So you can use the timestamps from events to associate them with timestamps from the optical flow maps or images or events from the other event camera etc.

As an example from the optical flow timestamp file zurich_city_01_a_optical_flow_forward_timestamps.txt

# from_timestamp_us, to_timestamp_us
55609607642, 55609707644
55609707644, 55609807643
55609807643, 55609907641
55609907641, 55610007596
55610007596, 55610107626
55610107626, 55610207629

The first optical flow map describes the per-pixel displacement from time 55609607642 to 55609707644. For EV-FlowNet we used the events between these two timestamps and regress the pixel displacements. This is arguably one of the most straightforward approaches.

[Hazeliii]

Thanks for your response.

But I am still confused. As an example from optical timestamp file thun_00_a/flow/forward_timestamps.txt

`
from_timestamp_us, to_timestamp_us
49599300523, 49599400524
49599400524, 49599500511
49599500511, 49599600529
49599600529, 49599700535
49599700535, 49599800517

`

I can see the time interval is about 100ms as in the code.
But the timestamps of events in the thun_00_a_events_left/events.h5 are numbers like this, which I don't understand
events_t[:100]: [ 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 10 10 10 10 10 10 11 11 11 11 11 11 11 11]

Is a set of events generated from eventslicer exactly corresponding to an optical flow map(the .png image) in the thun_00_a/flow/forward ? And the optical flow maps are give as 2/4/6/8.png . Are they generated associated to every second one in a group of ['event_volume_old', 'event_volume_new'] ?

[magehrig]

• "But the timestamps of events in the thun_00_a_events_left/events.h5 are numbers like this": You have to add t_offset from the h5 file to get the correct timestamps (as described here https://dsec.ifi.uzh.ch/data-format/). This is done automatically if you use the eventslicer class.
• "Is a set of events generated from eventslicer exactly corresponding to an optical flow map(the .png image) in the thun_00_a/flow/forward": Only if you provide the from_timestamp_us and to_timestamp_us to the eventslicer function get_events.
• "Are they generated associated to every second one...": Yes, exactly.

[magehrig]

Let me know if your questions have been answered

[Hazeliii]

Thanks for your help. But I have't solved the problem yet.
As an example from thun_00_a , the first line of the timestamp.txt of the flow map is :

49599300523, 49599400524

As you said, I changed the from_timestamp_us and to_timestamp_us to the eventslicer function get_events. acording to the timestamps of flow map, as follow, I am trying to fetch events whose timestamps are within [49599300523, 49599400524]:

`

timestamp_file = flow_dir / 'timestamp.txt'
assert timestamp_file.is_file()
file = np.genfromtxt(
timestamp_file,
delimiter=','
)
self.start_timestamps = file[:, 0]
self.end_timestamps = file[:, 1]
... ...
ts_start = [self.start_timestamps[index] - self.delta_t_us, self.start_timestamps[index]] # [old, new]
ts_end = [self.end_timestamps[index] - self.delta_t_us, self.end_timestamps[index]]
event_data = self.event_slicers.get_events(ts_start[1], ts_end[1])
`
But I didn't get the same number of events when using 't' and other features(p/x/y) to get events:

`
events['t'] = time_array_conservative[idx_start_offset:idx_end_offset] + self.t_offset
print('events[t]:', events['t'])
for dest_str in ['p', 'x', 'y']:
events[dest_str] = np.asarray(self.events[dest_str][t_start_us_idx:t_end_us_idx])
print('events[dest_str].size:', events[dest_str].size)
print('events[t].size', events['t'].size)
assert events[dest_str].size == events['t'].size
return events

`
The result is :

I didn't change the code except for the t_start_us and t_end_us parameters of the eventslicer function get_events. . And I really don't know why.

[magehrig]

I cannot replicate your problem. I downloaded https://download.ifi.uzh.ch/rpg/DSEC/train/thun_00_a/thun_00_a_events_left.zip
and set the path to the file thun_00_a_events_left/events.h5 in the following script. I get 804964 length for all the arrays t, p, x and y.

from pathlib import Path

from eventslicer import EventSlicer
import h5py


if __name__ == '__main__':
    event_path = Path('/path/to/events.h5')
    assert event_path.exists()

    t_start = 49599300523
    t_end = 49599400524

    with h5py.File(str(event_path), 'r') as h5f:
        slicer = EventSlicer(h5f)
        events = slicer.get_events(t_start, t_end)

    print(events['t'].shape)
    print(events['p'].shape)
    print(events['x'].shape)
    print(events['y'].shape)

Please also execute this script with the same file and let me know about your results

[Hazeliii]

I downloaded the data set again and used your script, and the problem was finally solved. Maybe there were some mistakes I could't find in my previous job. Thank you very much for your help.
Best wishes.

[JamesYang110043]

I cannot replicate your problem. I downloaded https://download.ifi.uzh.ch/rpg/DSEC/train/thun_00_a/thun_00_a_events_left.zip and set the path to the file thun_00_a_events_left/events.h5 in the following script. I get 804964 length for all the arrays t, p, x and y.

from pathlib import Path

from eventslicer import EventSlicer
import h5py


if __name__ == '__main__':
    event_path = Path('/path/to/events.h5')
    assert event_path.exists()

    t_start = 49599300523
    t_end = 49599400524

    with h5py.File(str(event_path), 'r') as h5f:
        slicer = EventSlicer(h5f)
        events = slicer.get_events(t_start, t_end)

    print(events['t'].shape)
    print(events['p'].shape)
    print(events['x'].shape)
    print(events['y'].shape)

Please also execute this script with the same file and let me know about your results

Hi @magehrig
Thank you for your nice work.
Regarding your response, I have a question.
As mentioned in your explanation: the timestamps t_start and t_end of the events come from the timestamp of ground-truth flow, does this mean that delta t is not needed, and it also implies that delta t is not fixed?

[magehrig]

@JamesYang110043 I am not sure what you mean with delta_t. Can you explain what you mean by that?

[JamesYang110043]

@magehrig

E-RAFT/loader/loader_dsec.py

Lines 295 to 300 in c58ce05

	def get_data_sample(self, index, crop_window=None, flip=None):
	# First entry corresponds to all events BEFORE the flow map
	# Second entry corresponds to all events AFTER the flow map (corresponding to the actual fwd flow)
	names = ['event_volume_old', 'event_volume_new']
	ts_start = [self.timestamps_flow[index] - self.delta_t_us, self.timestamps_flow[index]]
	ts_end = [self.timestamps_flow[index], self.timestamps_flow[index] + self.delta_t_us]

In thedef get_data_sample function within theSequence class for testing case, I noticed that it subtracts delta_t_us from a given timestamp to go back in time and adds delta_t_us to go forward in time. This means that[timestamp - delta_t_us, timestamp]corresponds to [from_timestamp_us, to_timestamp_us], and similarly [timestamp, timestamp + delta_t_us] represents the next [from_timestamp_us, to_timestamp_us]. Is my understanding correct?

In the training case, from_timestamp_us and to_timestamp_us are come from the timestamp of ground-truth flow. Therefore, if I want to replicate get_data_sample for the training case, I wouldn't need delta_t_us, correct?

[magehrig]

Partially correct. There are two set of events / event representations that are extracted by this function:
First, the events within {t_{k-1}, t_k} for the first feature map representing time t_k.
Second, the events within {t_k, t_{k+1}} for the second feature map representing time t_{k+1}.
The flow gt is from t_k to t_{k+1}.
You need both set of events for E-RAFT even during training. Also you need delta_t_us to compute t_{k-1} from t_k.
I hope that makes sense.