General Information and Work in Progress

[Cleric-K]

BlackboxToGPMF is a "work in progress" solution, so many problems and questions will arise.

Let this discussion be the place to ask questions, share and contribute to the project.

It is a continuation of what started at ElvinC/gyroflow#1

Currently the active branch of the project is https://github.com/Cleric-K/BlackboxToGPMF/tree/gui
It is entirely developed by @attilafustos

[Cleric-K]

Here I'll repost what was initially posted at ElvinC/gyroflow#1

Here are the scripts gopro.zip.
They are very chaotic because it was all experimenting, just trying to make it work.
To be honest, I haven't looked at them in months and now even I have trouble figuring out some of the parts :D

Some of the scripts were actually stand alone tests and have no relation to the overall project but I have left everything in.
In general, gp.py was supposed to be the main script. It takes few cmd line args:
1/ the bbox exported as csv from bbox explorer
2/ the input mp4/mov file
Here we have two cases: if we sync using only one point
3/ is the offset as seen in bbox explorer
If using two points for sync, first we sync at the beginning of the log:
3/ offset
4/ the time at the cursor
Then we sync again near the end of the log
5/ offset
6/ the time at the cursor

If both offsets are equal it means the video and fc clock are ticking in sync and there's no
need for 2-point sync.

The general logic is as follows:

1. First step is to correct the FPS. GoPro produces files with strange time base like
   30000/1001 or 60000/1001. RSTG expects num/denom like that so we have to switch the time base of
   our file if needed. It does not modify the video data itself, only the numbers in the header
   which basically tells how long a single frame from the video data should last. Obviously
   this will put the video and audio progresivelly out of sync but it is not a problem because
   at the end we can restore the original num/denom.
2. Then we read and parse the bbox csv. Only interested in the time (useconds) and the gyro columns. The time column measures
   time from the moment the FC is booted (not from arming/bb start) that's why we subtract the first time value from every time value in order to get zero based microseconds time.
   It is important to measure very accurately the camera tilt of the HD camera because gyro data has to be rotated.
3. Using the offsets provided as args we map the time column from FC time to video time (bbox.map_time)
   If using single point sync we simply add the offset to every value of the FC time (btw it's using numpy so the t in
   map_time is actually numpy.array).
   If using two points, it is a matter of a simple linear mapping. We know what the offsets should be in time1 and time2 so
   we simply make linear function that interpolates the offset from time1 to time2 and adds the interpolated offset at
   every point.
4. Now we have vectors with timestamps corrected for video time and corresponding gyro data.
   Next step is to get the gyro data at every GPMF frame we'll generate. The timestamps of the bbox gyro data, although
   now in the correct time base, have different frame rate than the needed GPMF samples so can't be used directly. For
   this we use numpy.iterp. We create linspace for the timestamps of the gyro GPMF frames and sample our
   timestamps/gyro data at that times. As a result we have equally spaced samples of the gyro data at the correct GPMF frame rate.
5. Then we construct the actual GPMF track and build the MP4 container.

Besides the GPMF track there's some additional meta data (in the beginning of the mdat section if I remember correctly).
RSTG parses video width/height/fps from the mp4 container, but it reads something in this meta data to know in what mode
the video was shot - basically it cares if Superview was used and the model ID of the camera.
For this part I simply have ripped that meta part from real GP Hero Session 5 files and I put it
into the output container.

Since RSTG is written in Unity it is quite easy to dig inside it using dnSpy and UnityAssetsBundleExtractor. After the basic video data is read, the correct preset is chosen among these:

If the source video is 1920x1080 for example the wide meta data should be put inside the file. For 4:3 - linear. The super preset could be useful only if working with older GoPro (like H4) that has superview but does not record gyro (this starts from 5).

For example, for my video footage with Runcam 3 I have used the H5S.1920.1080.wide.60000 preset. You are quite correct that RSTG has database for the lens geometry (in the different modes) of different models:

So yes, the closer the lens of our camera is to that of gopro - the better. For example if I work on my runcam3 footage with the superview preset the results, as it can be expected, are very distorted. But it seems that the wide preset looks more or less OK.

gp.py dumps the meta at the end as meta2 file. I used to manually convert that to meta.js using the bin2array.py script. Then with viz.html it was possible to make basic check if the transformed gyro data makes any sense. The html parses the meta data and gets the GPMF data, and draws basic symbol over the video to indicate the gyros.

I have written my own routines for mp4 and gpmf. It turned out that they are quite clear formats so using external libs was not that necessary and furthermore I had to do it in order to understand better their structures.

The virtualGimbal project is interesting. In what format does it expect the gyro data? Wouldn't it be possible to convert our bbox data to their format?

If I have missed something, please write back and I'll try to explain further.

Best~

[spinorkit]

I just discovered your work, it looks very exciting! So with this lens data one could write a filter for, say VirtualDub, to correct e.g. a Runcam lens to match a gopro lens before running the video thru reelsteady so it could do the best possible job?

[Cleric-K]

It should be possible, although I'm not very sure how exactly this should be done. Probably some reference grid should be filmed with both cameras in exactly the same way and then, based on that, a transformation should be derived that can warp the runcam image to look like gopro's.

But I think it should be tried to run the runcam footage directly in rsgo as gopro WIDE mode. From my experience distortions were not something I could detect visually so it puts into question whether the whole lens transform operation is worth the effort. Any transformation adds additional loss of quality so it should be decided if the reduction in distortion outweighs the loss of quality.

But nevertheless, it will be an interesting tool if it can be done.

[spinorkit]

Yes. In the (distant) past I have used VirtualDub filters to stabilize footage from a Mobius cam. Without correcting for the lens distortion the shape of the landscape would fluctuate noticeably as the image was shifted to stabilize it (this was very bumpy footage from a wing). I did find code to correct the Mobius lens and put that in a filter, which I inserted before the stabilization filter. This helped, but results were nowhere near as good as RSTG.

[Cleric-K]

Asked at ElvinC/gyroflow#1 (comment)

One more question, I get asked a lot on the groups: how to use the offset1, offset2, time1, time2 parameters.
In my demos I only used time 1 parameter with 0:0 because the length of gyro was the same as video length.

Two point sync is needed when the the video time ticks at different rate compared to blackbox time.

It can happen that we sync very well our video somewhere at the beginning of the log (we adjust the offset in bbox explorer and the gyro curves match perfectly the video movements) but then go to the end of the log we see that gyro and video are out of sync.

For this reason we should write down the offset and the time of the log, where the sync is OK (in the beginning of the file). Then we go at the end of the file and sync there. The offset will be different (now if we move to the beginning we'll see that we are out of sync there). Now we write down the new offset and the new time where this offset is correct.

Fore example:

We go to time 0:9 and adjust the offset of the video to make perfect sync between video and gyro - for example +0.533.

These are the first two value.

Now if we go towards the end of the log, we'll find out the video and gyro are no longer in sync. We adjust the offset again to match them and see that, for example, at time 1:14 we have offset +2.343. These are the second two values.

Using these four number we can interpolate linearly, so that between log time 0:9 and 1:14 the video offset will go from +0.533 to +2.343.

[uLightMe]

I'm a java developer by day. This project really peaked my interest, I would love to use real steady go on my dvr footage from caddx vista or from my old session4. I was wondering if there is some way I could contribute to this project. If there is a better place to ask please point me to it.

[Cleric-K]

Currently @attilafustos is the one working on the project. Because of lack of time, I'm currently out of sync with what's going on the active branch.

Maybe also check his FB Group.

[xafoHD]

Hi, I have a question. When I start the exe, there is an error. It says: failed to execute script. I hope someone can help me.

[t0ffi]

you need to run python script directly

[t0ffi]

https://youtu.be/fPx5X5xP4Vc

[xafoHD]

All right and thanks. Now I can run the program, but there is an error when I try processing the Video. It says: "struct.error: unpack requires a buffer of 8 bytes" Can you help me with this too? I would be very thankful.

[t0ffi]

I occur same problem when trying to concat 2 files using ffmpeg and also waiting for someone to help me.

[QuigonLuk]

Hi, i 've a problem.
i want to use on reelsteady a goproh6black's mp4...but i've recorded in linear mode so RSgo don't support it. i've try to use GP.exe but the .csv that i've extract from the mp4 file have a different syntax.

GP.exe don't give me error but it don't work

do you have any suggestions for me?
thnx

[attilafustos]

The current implementation only supports Betaflight Blackbox CSV file.

[QuigonLuk]

unfortunately I know... i'll try to convert the file in the corrtect syntax

[walldodge]

Same thing, blackbox log converted and script didnt work for me.

[noobtoob4lyfe]

I'm having the same issue where it's stuck on the "original FPS" part. Was anyone able to solve this?

[All4Gis]

Hi, I'm trying to insert klv telemetry into a video
https://gist.github.com/All4Gis/509fbe06ce53a0885744d16595811e6f
and I think you have this goal resolved, that's great, but could you share me the result video, any examples?Only the result video to check the video streams

[starline]

Hey! trying to insert gyroscope data into a video using a program. But the data does not change at the output. Help me to understand. The program does not issue errors, it creates files. But the gyro data doesn't change