arbitrary-image-stylization-v1-256

[Jonathhhan]

I try to run the arbitrary-image-stylization-v1-256 model: https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2
I can load the model without any errors, but it chrashes if I try to run the model (without an error message).
This is how I try to run it:

	if(!model.load("magenta")) {
		std::exit(EXIT_FAILURE);
	}

	model.setup({ {"serving_default_placeholder:0"} ,{"serving_default_placeholder_1:0"} }, { {"StatefulPartitionedCall:0"} });

	auto input_1 = cppflow::decode_jpeg(cppflow::read_file(std::string(ofToDataPath("example_1.jpg"))));
	input_1 = cppflow::cast(input_1, TF_UINT8, TF_FLOAT);
	input_1 = cppflow::expand_dims(input_1, 0);
	auto input_2 = cppflow::decode_jpeg(cppflow::read_file(std::string(ofToDataPath("example_2.jpg"))));
	input_2 = cppflow::cast(input_2, TF_UINT8, TF_FLOAT);
	input_2 = cppflow::expand_dims(input_2, 0);
	std::vector<cppflow::tensor> vectorOfInputTensors = {
	 input_1, input_2
	};
	auto output = model.runModel(vectorOfInputTensors);

Any idea what I am doing wrong (or if the model runs at all with ofxTensorflow2)? Thank you.

[bytosaur]

Hey @Jonathhhan,

I am currently on vacation and can only look into it at the mid of next week.
It looks alright how you handle the input. Maybe you are running out of Gpu memory? Try setting memory growth with

if(!ofxTF2::setGPUMaxMemory(ofxTF2::GPU_PERCENT_90, true)) {
  ofLogError() << "failed to set GPU Memory options!";
}

[Jonathhhan]

Hey @bytosaur,
I think I found a solution and made an example: https://github.com/Jonathhhan/ofxTensorFlow2/tree/universal-sentence-encoder/example_arbitrary-image-stylization
The advantage is (in my opinion) that you just need to load an image and not a model for the style.

[bytosaur]

Yup arbitrary style transfer is superior in that sense. How much fps do you get and what hardware do you use?
The same model has been used in Alexander Schubert's crawl3rs ;)

[Jonathhhan]

It runs at 60FPS and 1280 x 694 (video resolution) with an RTX3090, but it needs a lot of power (around 340w). Not sure how to reduce that. With 1 FPS it goes down to 120 w.

[bytosaur]

hey @Jonathhhan,

thank you for your contribution! I am currently trying out your example on an RTX 3080.
I have two ideas how to improve the performance:

• use multiply by 1/255.0 instead of div by 255

   input2 = cppflow::mul(input2, cppflow::tensor({ 1.0f/255.f}));

• allocate the image beforehand instead of for every image

   imgOut.allocate(videoPlayer.getWidth(), videoPlayer.getHeight(), OF_IMAGE_COLOR);

Please feel free to make a PR for this example!

BTW could you please name the example example_style_transfer_arbitrary? This way they are next to each other :) and i would like to keep the basic example on top

[bytosaur]

oh and please keep in mind to use the standard OF_ROOT. I.e. leave the variable commented in config.make ;)

[Jonathhhan]

hey @bytosaur,
thanks for the improvement ideas. mul instead of div does not seem to make a noticable difference and i need to allocate the output image after running the model, because sometimes the output shape differs slightly from the input (in my case the width difference is 2px).
Here is a branch with just the example. I changed the name and made some small changes:
https://github.com/Jonathhhan/ofxTensorFlow2/tree/example_style_transfer_arbitrary/example_style_transfer_arbitrary
I also tried to adapt the example_style_transfer example but I think I had problems to setup the model in the model class, so it didnt work (but in theory it should ;)).
And I copied config.make from the other example, cant remember that I changed something (also not sure how it works, exactly).
I do not know how to keep the empty folders in github with the help of gitkeep files, they always disappear (there should be a data/model folder)...
And the readme is not written yet :)

[bytosaur]

Hey,
Yes the mul doesnt make much of a difference in compute time but I still prefer it actually. Division just give me the chills :D
The difference in pixels sounds really really odd... It worked pretty well for me and improved the performance quite a bit.
No worries about using inheritance. I think it s good to have some examples that are more straight forward.
I'd be happy if you add a little README (please also put in your and the models reference :)).
Then we can merge the example and make our small and more stylistic changes ;P
Never had any issue with leaving and empty .gitkeep file in the folder... are you sure you added it to the commit?

[Jonathhhan]

Alright :) It seemed that the video had a strange resolution or was corrupted, with another video it works without allocating from the shape. I also replaced div with mul. And I read the video texture directly into float pixels now, so no further conversion is needed. I will have a look into the other things...

[bytosaur]

hey @Jonathhhan,

i have already pushed a branch with some smaller changes. I ll first adjust and merge your other examples before modifying the download scripts :)

[Jonathhhan]

hey @bytosaur,
thanks a lot for improving and polishing the examples (and for the addon itself, of course). I am still a beginner and already learned something from it.
I also found some small things that I could have done better (but nothing with a big increase in performance).
I can do some pull requests from your branches.
This way for example, the ofFloatPixels object is not needed anymore:

		input = ofxTF2::pixelsToTensor(videoPlayer.getPixelsRef());
		input = cppflow::expand_dims(input, 0);
		input = cppflow::resize_bicubic(input, cppflow::tensor({ 360, 480 }), true);
		input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
		input = cppflow::div(input, cppflow::tensor({ 255.f }));

[bytosaur]

hey @Jonathhhan,
we highly appreciate your examples! I didnt do much on the code side of things except for packing the inference part into a separate function. Your examples have been pushed to this branch and will be merged soon :)

[bytosaur]

included in #29