Negative pixel values after applying sr.transform()

[bastianmorath]

Thanks a lot for effort you put into this library. I have two questions which I hope you can give some insight into:

Negative Pixel Values

I realised that often after applying the transform() method to the image, the resulting image contains negative pixel values.

My pipeline looks as follows:

1. I have a reference portrait image where I want to paste a distorted face of the same person onto it
2. I pad the face image with a black border to make it the same size as the reference portrait image (Roughly putting the face at the same position as the reference portrait image to make the algorithm more robust)
3. I split the portrait image into the 3 separate rgb channels, register the two images by using their grayscale-version, then applying the resulting transform onto the three channels. After that I stack the separate channels together to get the final image.

Unfortunately this gives me negative values.

Is there something fundamentally wrong with my approach?

transform_stack glitches

When doing the above mentioned approach with multiple distorted faces on the same reference portrait image, it produces nice results when processing each frame separately.
But when using the register_stack and transform_stack() methods, the resulting frames have glitches such as black lines appearing at random positions or weird transformations (Please see the attached screenshot). Do you have any idea why this might happen?

[glichtner]

Hi Bastian,

would you be able to provide a small working example (code + data) for both your issues for me to reproduce and research the causes? That would be very helpful.

Gregor

[bastianmorath]

Hi Gregor,

Thanks for the fast response.
Please find attached a minimal working example that shows the two problems.

Bastian

pystack.zip

[glichtner]

Hi Bastian,

The main issue here is that your stacks have the frames in the third dimension (width x height x frames), whereas pystackreg by default assumes the frames to be in the first dimension (i.e. frames x width x height). Thus you have been registering the data from the height x frames dimension and then looked at the original image data (width x height), where then black lines etc. are visible (because you are not looking at the transformed dimensions).

What you need to do is to change your code to explicitly state the axis of the frames:

sr = StackReg(StackReg.TRANSLATION)
sr.register_stack(mov_grays, reference='first', verbose=True, axis=2)

out_r = sr.transform_stack(r_channel_frames, axis=2)
out_g = sr.transform_stack(g_channel_frames, axis=2)
out_b = sr.transform_stack(b_channel_frames, axis=2)

Unfortunately, there seems to be a problem when using the transform_stack function with another axis than the first (i.e. axis=0), at least in my setup. I have tentatively fixed this (and added a function that raises a warning when the selected axis might not be the "time" axis), but I aim to add example data and tests before I will release the next version.

You therefore have three options:

1. Clone the latest commit from this repository and install from source:

git clone https://github.com/glichtner/pystackreg.git
cd pystackreg
python setup.py install

2. Manually transform the stack using the transform() function.
3. Reorder your image dimensions so that frames are the first dimension (e.g. using numpy.moveaxis)

Let me know if that helps.

Gregor

[glichtner]

As for the negative values: This is exactly the behavior implemented in the original TurboReg ImageJ plugin and therefore intended for this python package. I leave it therefore to the user to decide what to do with negative values (e.g. clipping as you are doing), at least for now.

[bastianmorath]

Hi Gregor

Thanks a lot for your help and the quick fixes, those indeed seemed to fix the aforementioned problems!

BTW, since I wanted to transform my frames according to the first frame (thus having to add this one to my transform-stack), I had one additional frame in the registering stack compared to the transformation stack, which produced the exception "Exception: Number of saved transformation matrices does not match stack length". To solve this, I just duplicated the first frame of the transformation-stack. As a suggestion, maybe in addition to using the 'first'- argument, it would be nice to also have the option to add a separate reference image as an argument to the .register_stack method?

I really appreciate the effort, keep it up!

Happy holidays,

Bastian

[glichtner]

Hi Bastian,

I have released a new version of pystackreg that should work when transforming a stack with a different axis parameter (other than the default value 0) and it should also raise a warning when potentially using the wrong axis.

As for your feature request, that could be a good idea, but I haven't had the time to implement it now. Feel free to implement it and open a pull request.

Best,
Gregor