-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optical Flow Definition #4795
Comments
I have the same question and look forward to answering it. |
To save you some time, here's what i found out so far:
To summarize, my visual & numerical tests show the following:
I'm looking forward to your input. |
This comment has been minimized.
This comment has been minimized.
I think this assumption are right. I have tested the assumption of using the simulated optical flow as the forward flow (t->t+1) or the backward flow (t->t-1) using the backward warping function provide by mmcv (https://github.com/open-mmlab/mmcv/blob/master/mmcv/video/optflow.py#L143). For given three rgb image rgb0_image I first normalize the raw flow data to the pixel displacements.
flow_10 flow_12 Then use the backward warping function. For forward flow, warp on the next image. For backward flow, warp on the pervious image.
The warped result using backward flow seems more matched to the frame In addition, the flow at time t and the image of at time t are simulated at the same time. At this time, the image of frame t+1 has not been simulated, and the future motion is uncertain. Therefore, I think it is impossible to obtain the forward optical flow from t to t+1 during real-time simulation, which is also consistent with the simple experimental results above. I'm not sure whether this conclusion is correct. I hope we can get further official clarification from the Carla team. |
Hey, @danqu130 thanks for the great visualizations! I have a question about how you obtained the |
The code I use to convert carla.Image flow_data = np.array([(pixel.x, pixel.y) for pixel in data], dtype=np.float32)
flow_data = flow_data.reshape((data.height, data.width, 2)) |
I was confused when I observed the simulated optical flow. It has obvious layered optical flow in some scenes, but it seems unreasonable. This is the simulated flow and the corresponding first frame. And this is the optical flow computed by RAFT (https://github.com/princeton-vl/RAFT-3D) using the first and the sceond frame. I calculated the EPE error of these two flow, and marked the position where the error is greater than 1 pixel as white. The layering problem in the CARLA flow does not match the real motion, which is also confirmed by the results of RAFT. I have tried a variety of motion speeds and simulation frame rates, which also have this problem. In addition, I have checked my optical flow visualization steps. Even if I directly use the method provided by Carla ( Hey, @milmario @jonasdieker, did you also encounter this? Do you think it's normal? |
@danqu130, under which simulated conditions (light, speed, camera intrinsic,...) do you get these artefacts? Can you please share your overall config for the above result? It seems that other carla-flow samples look quite good, so I am wondering where it comes from. Just to make sure: Is this the (very) first frame you are showing? If this is the case, it would make sense to have such artefacts, since OF is backward. |
CARLA version is 0.9.13, 4.26.2-0+++UE4+Release-4.26 I think it is not normal for such artifacts to appear in this autonomous driving scene, whether forward flow or backward flow. In addition, has the CARLA team confirmed that the simulated optical flow is backward flow? |
@danqu130 I've encountered the same issue of this staircase effect. Did anyone solve it? |
Unfortunately, I don't have any more updates. |
Hello, I'm not sure that Carla Optical Flow is in backward mode, I made some test that give me the impression that there are forward. Just by recording a car passing through the camera: You will see that the car on optical flow image (middleburry colormap) is aligned with N+1 image and blue color correspond to a vector moving to the left: This test make me think that we are on forward mode but a confirmation of CARLA team could be interesting. I also noticed the quantization phenomenon and the 10 bit explanation from this ticket : #5514 seem interesting but same, some confirmation from CARLA team. Does anyone find a trick to workaround this effect ? @Axel1092 do you have any clue ? Best regards |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
For me, the definition of the optical flow output of CARLA is not clear.
The text was updated successfully, but these errors were encountered: