New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert to TF Lite: GAN model to cartoonize photos #9
Comments
Thank you for bringing this to my notice, @khanhlvg. I have self-assigned this and will start working on it. I will keep you posted about the progress here in this thread. |
Hey @khanhlvg I have started working on it. I have been able to extract a
Here's the Colab Notebook. |
The stylized image looks really cool! Could you try converting with tf-nightly? The model has a dynamic shape input and it's only supported since TF 2.3. |
Ah bummer. How could I miss out on that part? 😖 I just tried it out and guess what? The TF Lite model (6.4 KB) is converted 😄 Now, what's needed is the pre-trained weights for the generator network. I have filed an issue already in their repository. As soon as I have an update, I will keep this thread posted. Here's the Colab Notebook made so far. |
Wow 6.4KB doesn't sound very right. I suspect that some weights might have been left out when converting to TF Lite. Could you try running inference with the TF Lite model? |
@khanhlvg yes, I suspected that one too. I tried replicating the original inference pipeline with the converted TFLite model and the output is coming to be spurious. Here's the Colab Notebook. Let me know if I am missing out on something. |
The issue came from your saved model export code. You use an intermediate output (
However I'm still facing an issue that the color of the output is off. I think there's an issue with pre/post-processing code but I haven't been able to pin point the issue. Can you take a look? |
I was able to make it working! Here is the notebook The weird color issue is because the model use cv2 for preprocessing and post-processing, which use BGR. However in your notebook, visualization is done with PIL which use RGB. Therefore, a small swap was needed. I run benchmark of the model on my Pixel 3 and it is incredibly fast! I think we can do live video cartoonization with this :)
|
Not sure if this is a coincidence again. I was just about to inform you about this @khanhlvg |
What next steps would you suggest @khanhlvg for the conversion? |
@sayakpaul
Besides, I made a mistake with benchmarking. I forgot to set the shape for the input image. The model is actually quite slow. For example, an 224 x 224 input on int8 model on Pixel 3 takes 729 seconds. So unfortunately it might not be suitable for live video.
|
Yes, even on a Colab CPU the model takes quite a while to spit out the results.
Sure. Could you mention which Metadata example I would refer to for this?
The model might not be very fast. Would we still want to do this? Now, of course, it's still a very cool example to create a mobile app with this kind of capability. |
It's because TF Lite runtime is not optimized to run quantized model on Linux. If you use float model, then it'll be using XNNPACK and it's blazingly fast. The quantized model runs faster on mobile devices vs. on Colab.
Please use the selfie2anime metadata format. It's the same concept: image in -> image out.
Inference time below 1s on CPU is still very good for still images. And the quantized model is just a little larger than 1MB so definitely it's very useful on mobile :) |
Yes, I am aware of this. Just wanted to bring the topic up so that you know. I will proceed with the next steps and post the updates here. Thanks! |
@khanhlvg a couple of updates. I converted four different variants: float16, dynamic-range, int8 and full int8. All of the models (metadata is yet to be added) reside here: gs://cartoon_gan (public). I did not export the float16 model to support dynamic shape as we had discussed. Here is the Colab Notebook that demonstrates the conversion and the inference processes. Observations:
That's all for now. I will get back to you after adding metadata. |
@khanhlvg metadata has been added to the models and they live here: gs://cartoon_gan/model_with_metadata. Here's the Colab Notebook. |
Thanks Sayak!
The model is trained with BGR images so if you feed it an image in RGB, the output cartoonized image won't look as good as if you feed it an BGR images.
Tensor index doesn't have any special meaning, just an id so feel free to ignore the actual number.
It's because the model cv2 loads images as BGR. Therefore if you load your input image using cv2, then you should do the I also tried running the fp16 model on GPU but faced a couple of issues. I'll ask the GPU delegate team to investigate.
Btw I'm taking days off until the end of this week so please expect some delays in my reply. Thanks :) |
Fair points, @khanhlvg. I will update the notebook accordingly. Could you verify if the metadata has been populated correctly when you get a moment? Upon your go-ahead I will proceed to TF Hub publication (which I know will be done after we are done with the GPU delegate issue). Also, enjoy your days off! |
@sayakpaul I should verify the tflite model (with metadata) in the Android app. Please place the tflite models with/without metadata under a ml folder here: https://github.com/margaretmz/CartoonGAN-e2e-tflite-tutorial/tree/master. Or update the README with links pointing to the models if you prefer to have them located elsewhere. It's difficult for me to hunt them down from the discussion thread here. |
I did not realize you created a repository @margaretmz as I was planning to have the notebook hosted from here. And once @khanhlvg and I were done publishing the models on TF Hub we the plan was to directly download them from TF Hub. You can retrieve the models (with metadata from here) :
|
@margaretmz FYI I went ahead and created a card for CartoonGAN Android App and I also created an issue from the card: #11. |
@khanhlvg the outputs with the Here's the updated Colab Notebook. |
@sayakpaul - Thanks for the links, I will try the tflite models with metadata in an Android app. |
Thanks, makes sense. I think we could work on a short document to enlist these different steps to let the interested contributors know about how we generally go about doing things here. Let me know your thoughts. |
Sounds good. How about use the wiki:https://github.com/ml-gde/e2e-tflite-tutorials/wiki/How-to-contribute-to-the-E2E-TFLite-Tutorials%3F? |
@margaretmz I think having it on README would work as well.
I am in 100% agreement. |
@khanhlvg I was able to fix the issue with int8 models. The problem was I was passing randomly created data of So, here's what I did: IMG_SIZE = 224
images_list = os.listdir('/content/source-frames')
# int8 quantization requires a representative dataset generator
def representative_dataset_gen():
for image_path in images_list:
image = cv2.imread(os.path.join('/content/source-frames', image_path))
image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))
image = image.astype(np.float32)/127.5 - 1
image = np.expand_dims(image, axis=0)
yield [image]
model = tf.saved_model.load('saved_model_dir')
concrete_func = model.signatures[
tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
tflite_model = converter.convert()
open('whitebox_cartoon_gan_int8.tflite', 'wb').write(tflite_model) The output is now coming as expected: The Colab Notebook is also updated with these changes. @margaretmz I have updated these models with Metadata as well but the links I earlier mentioned in this comment will still remain the same. Looking forward to this exciting project :) |
Thanks @sayakpaul. I will add Android code to [the project repo](https://github.com/margaretmz/CartoonGAN-e2e-tflite-tutorial soon. So far I'm encountering errors running inference with the tflite models:
@khanhlvg - do we also need to add OpenCV library to Android in order to have the image as BGR? |
@margaretmz are you using the FYI, the output image also needs a bit of processing as you'd see in the Colab Notebook. If I missed out on that in the while populating metadata (Colab Notebook), please let me know. FYI, I had created a separate issue thread for the Android application demonstrating this model. Also, were you able to import the models via Model Binding? |
I think this is because the ML Model Binding doesn't support resizing the input shape yet. Let me file a bug to the Android Studio team. Thanks for surfacing the issue!
As model interface generated by ML Model Binding takes Bitmap as the input, it needs to support models with BGR input first. Let me file another feature request to the Android Studio team. As Android Studio has long release cycle, I think it may take months before the before went into beta. Shall we start building the demo directly with TF Lite interface (e.g. using TF Lite Interpreter and Support Library for pre-processing)? |
I think this would be a win-win. We already have the dynamic-shape supporting models with us. If we decide to go down this path what shape would y'all suggest we keep?
Might be a better choice. |
If we create a fixed shape model, then (512, 512) is probably good enough. |
Fair enough. Once we all decide the next steps from here I can go ahead and create separate models with fixed-shaped inputs and populate them with metadata. |
Yes I'm using tf-nightly interpreter
If the metadata is set properly the post processing, it should be taken care of in the generated class.
I will try to track the issues under the project repo to that's it's easier to find. Let's try to use the issues in this repo for assigning project resources and then track details under each project repos.
Yes, I'm able to import all 3 models.
Fixed shape model to proceed sounds good to me, too
It will take more time but I will give it a try. |
Thanks, @margaretmz. I will proceed with converting the models with fixed shaped (520, 520) inputs.
I am in agreement. I am going to close this issue, and let's shift all of our communication here. Cc: @khanhlvg |
@khanhlvg and @margaretmz models published on TF Hub: https://tfhub.dev/sayakpaul/lite-model/cartoongan/dr/1 |
This model is implemented in TF 1 and can convert photos to cartoon. It looks quite visually impressive!
https://github.com/SystemErrorWang/White-box-Cartoonization
@sayakpaul
Would you be able to try converting this model to TF Lite?
The text was updated successfully, but these errors were encountered: