Request: offer image-enlarging library that uses AI (TensorFlow)

[AndroidDeveloperLB]

System information

• TensorFlow version (you are using):
  tensorflow-lite-2.3.0
• Are you willing to contribute it (Yes/No):
  Not sure how.

Describe the feature and the current behavior/state.
https://issuetracker.google.com/issues/176311044

Will this change the current api? How?
No.

Who will benefit with this feature?
Everyone who wishes to enlarge/enhance images.

Any Other info.
I know we already have a sample of it, but it's very restricted (only allows from 50x50 input images) and isn't comfortable at all to use on new projects.

[jdduke]

isn't comfortable at all to use on new projects.

Can you be a bit more specific? Is it that you'd like to see a higher-level task-like API for image super-sampling?

[AndroidDeveloperLB]

@jdduke Specific about what? You mean which sample? If so, this:
https://github.com/tensorflow/examples/tree/master/lite/examples/super_resolution

Check the link. I've explained there more.
I think it could benefit plenty of apps to have a nice library that helps with resizing/enhancing images.

[lu-wang-g]

By reading https://issuetracker.google.com/issues/176311044, I think this user request can be mapped to our existing projects by:

1. Adding a new Model Maker API that customizes the super resolution training script and allows configuring input/output image size. A stretched goal, as mentioned in b/176311044, will be wrapping the script into some UI tool, such that users can simply update training images and get the custom model.
2. Adding a new Task library API that can consume the generated super resolution models.

@AndroidDeveloperLB what do you think? Is it something you're thinking about?

[AndroidDeveloperLB]

@lu-wang-g
There are various online solutions for enhancing images, but they cost money and most of them indeed run on the server instead of locally :
https://www.youtube.com/results?search_query=image+enhance
https://www.google.com/search?q=enhance+image&oq=enhance

I want a free solution, that will run directly on the device, no server needed.
Not sure what you suggest. I offered multiple ways to have such a thing.

In the end, for most developers, a simple usage could be available, as adding just a dependency:

    implementation 'com.google.tensor-flow:super-resolution:1.0.0'

(or the core, with addition of some small extra steps)

But, it could be configured to various purposes, to reduce the size that is added to the app or to focus on specific sizes and contents, based on the needs.

Today we have nothing. We have to either use the sample we've found of "super resolution", and only resize from 50x50 pixels, or we have to train images ourselves, and still probably be restricted to some width&height as the input. Or, as I've mentioned, we need to use some third party solution that is almost always server based or paid for.

I requested :

1. A more general solution, not restricted to input size of the image (not to force us to use input image of specific resolution), that will fit for most cases that developers would love to have.
2. A solution with minimal effort from developers - won't require knowing how to train an AI. At most, we could just give it the images to train with.
3. Since we might be dealing with mobile devices, where storage is important, think of a way to reduce it. Maybe work with Play-Services that will have a model file that is used for all app? Maybe ask us what are the specifications to create the file, and let us download it? Maybe we could tell it "OK use up to X MB for Y purposes" (Y can be "animals", "people", or "general")...
4. Free. The device is doing the work, so no need for costs.

I think now it's the time to do it. AI is gaining more and more popularity, and devices are quite capable of doing this job without a server side.

I think there are plenty of ideas you can come up with to help developers use such a functionality.

[lu-wang-g]

As you described, Google MLKit will be a more ideal solution for the use cases. MLKit provides turnkey solutions and users don't need to worry about any underneath ML technologies. TFLite currently focuses more on on-device ML development and customization, so some basic background of ML is required. I agree we should do more than just a sample super resolution app. In fact we are collecting user requests on the most interested tasks they want us to improve more, and super resolution was nominated quite a few times. Thanks for sharing your ideas. We'll gather all the feedback and prioritize on the most wanted cases. Please stay tuned on our future updates.

[AndroidDeveloperLB]

@lu-wang-g I don't know. Depends on the needs.
The sample is sadly even quite hard to convert to a library.
How can I vote for image-enhancing , and others?
How can I stay tuned about this?

[lu-wang-g]

You can create requests like this to vote for other cases. This issue can be used as a tracking bug to track any future updates. You can also follow our roadmap: https://www.tensorflow.org/lite/guide/roadmap, for the overall TFLite plans.

[AndroidDeveloperLB]

@lu-wang-g Where? Where do I vote? Where can I see other requests of others? Where can I subscribe to such requests?

[lu-wang-g]

By "vote", I mean you can file request like this and explain the use cases you're interested. And we'll collect all the requests and make plans accordingly. We're also planning to send out a user survey to this email group: https://groups.google.com/a/tensorflow.org/g/tflite. Please subscribe to it and we'll keep you updated there.

[AndroidDeveloperLB]

@lu-wang-g It was written "super resolution was nominated quite a few times." , so this means it was already requested, and that I can vote on it.
Where can I write and find about this?
Searching "super" , "resolution", "enhance" on the website doesn't return me such a thing.
Anyway, I made a new request there too:
https://groups.google.com/a/tensorflow.org/g/tflite/c/RLtObidqqpY

[lu-wang-g]

By saying "super resolution was nominated quite a few times", I actually mean we've heard several request from Google internal users and ML developers. I don't think you can search from them on the website. Feel free to let us know through github / tensorflow user group (they are identical) if you have other new use cases in mind.

[AndroidDeveloperLB]

@lu-wang-g But I couldn't see even one place that it was nominated.
What other things have people requested? Maybe from them I could think of other nice additions.

[pkgoogle]

Hi,

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

The TFLite team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the TFLite space.

Thanks.

[AndroidDeveloperLB]

Please explain further the answer to the request.
Do you mean that now we have this ability?