Use GPUImage to enhance scanned documents

[julianschiavo]

I think we could use GPUImage(2)'s AdaptiveThresholding to enhance the scanned image (It would be optional).

Example:

[Boris-Em]

I think that we want to limit the use of third party libraries. Could we look into implementing this feature using Apple's APIs?

[julianschiavo]

I believe it requires OpenCV or other frameworks, but i'll look into it.

[julianschiavo]

Update on this after I made a test implementation:

1. Adaptive thresholding requires OpenCV or GPUImage from what I can tell.
2. Adaptive thresholding can sometimes make the image worse, and should be optional
3. Adaptive thresholding might make sense as a hidden property passed to the host app, as it probably results in better OCR. On the other hand, the host app could just do it themselves.

[oferRounds]

@Boris-Em GPUImage is a great high-quality library, which is well tested. In my opinion we shouldn’t be too picky in using third-parties like this one

[jcampbell05]

In theory this could be done in Metal :)

[saormart]

@justjs, any update on this ? This will be added to the framework / and also in the example?
Thanks!

[oferRounds]

@jcampbell05 GPUImage 3 already uses Metal

[saormart]

Well I was able to make it by using Adaptive thresholding with OpenCV2. (that works for me so far)

[julianschiavo]

I've started work on a Pull Request that implements this, likely with GPUImage.

A few things:

• GPUImage3 does use metal, but it's barely done yet and is still missing a lot of functionality. As per the docs, however, we should be able to replace GPUImage2 with GPUImage3 relatively easily when it's time.
• I'm aware that 3rd party libraries can be an issue, but I do think GPUImage2 is a highly tested, reliable library that would greatly improve WeScan

I'll update this issue later if/when I get the basic functionality working.

[jcampbell05]

Perhaps it may just be enougth for WeScan to have hooks for users to extend for their own need. i.e allow user's to run their own preprocessing code for the image generation via a delegate callback. WeScan keeps 3rd party dependencies out of it's code but users can utilize GPU Image as needed to enhance what is shown in the cropping screen.

Could open door for other things like extending the scanning functionality by allowing access to each frame from the buffer (if say someone wanted user to scan QR Codes)

[julianschiavo]

🤔 While I do somewhat agree with that, I feel like getting the thresholding/enhancing right is something that most developers don't have time for, but would still be useful to most, if not all, people who implement WeScan.

Any other thoughts on this? For now I'm working on the PR as it's not too hard, we can discuss further later on if needed.

[Boris-Em]

Thanks for picking this up @justjs.
I don't think that using GPUImage is a good solution here.
One of our values is to keep WeScan as small of a dependency as possible. GPUImage is a huge library that does way more than what we need, even for this feature.

@jcampbell05, this is a better solution, but it would probably add quite a bit of complexity to the project both for us and our users. If we could avoid it, that'd be great!

Could we take a look at implementing this feature ourselves? It doesn't seem like it would be a lot of work.

[jcampbell05]

@jcampbell05, this is a better solution, but it would probably add quite a bit of complexity to the project both for us and our users. If we could avoid it, that'd be great!

Good point.

Do we have a list of what they do ? is it just a basic curve adjustment (like you would do in photoshop) ?

[Boris-Em]

Maybe this is a good start: https://homepages.inf.ed.ac.uk/rbf/HIPR2/adpthrsh.htm

Also: https://stackoverflow.com/questions/36184255/adaptive-threshold-cikernel-cifilter-ios
https://stackoverflow.com/questions/36287861/how-to-use-coreimage-to-create-a-smooth-threshold-filter

[julianschiavo]

Fair enough @Boris-Em. I've basically finished the GPUImage2 version as a quick proof of concept (it's on justJS:WeScan develop branch), so at least we can see what it looks like for now.

I'll see what I can do on the other implementations.

[julianschiavo]

If anyone wants to play around with it as a demo, I've got adaptive thresholding working with GPUImage2. Make sure to run git submodule update --init.https://github.com/justJS/WeScan/tree/develop

[julianschiavo]

Very happy to announce that I managed to get it working (way quicker than I expected) without GPUImage2! #80

[jcampbell05]

Lets close this :)

[julianschiavo]

Closed as added in #80