Similarity comparison on Frontend or Backend ? For optimum performance

[uzair004]

What I want build:

• I want to build a website where user upload an image, face-embeddings will be extracted from it and stored in JSON file along with it's source where image is stored.
• guest user upload an image, we will compare it with others images we have and return best or many similar images along with related data(details of person we have stored) with it from database. (Kinda image search)

**How **:
One way to go is do all comparison on node Backend and send response to Frontend (Backend handle all comparison)

My plan:
I am thinking to shift all comparison and similarity check load on individual user instead of Backend.
How:

• Frontend convert image to embedding
• request already recognized faces from Backend
• Backend send all JSON file data to Frontend (contain face embedding along with it's image source)
• Frontend compare it's own embedding with all sent by Backend
• Frontend get one or many result based on threshold.
• Frontend request Backend for images and it's associated data.
• Backend send required data to Frontend

Why:

• As NodeJS is single-threaded I don't want one user to hog entire process
• comparison operations will be shifted to each client instead of Backend handling them. (Performance)

The Problem:
Sending Huge amount of JSON data to Frontend:
Json file may have huge amount of data (hundreds of images processed) sending them to Frontend may impact performance of website, i.e rendering UI, or possible hung up browser.

Note: my first project so any suggestion will be welcomed.

[vladmandic]

As NodeJS is single-threaded I don't want one user to hog entire process

NodeJS is single-threaded, but you can use a preinitialized pool of worker processes and
send each request to worker with currently lowest load

See demo/nodejs/node-multiproces.js and demo/nodejs/node-multiprocess-worker.js for example

That way you can even say to user 'you're number xx in the queue` without ever blocking the server or the response to the client

Json file may have huge amount of data (hundreds of images processed) sending them to Frontend may impact performance of website, i.e rendering UI, or possible hung up browser.

In theory, you could break down compare operation so server only finds rough matches and then generates JSON of those to send to the client which then does in-detail processing.
Such 'rough match' function doesn't exist right now, but would not be difficult to write (I wouldn't add it to the library itself as it's a very specific use-case)

Also, best to process JSON in a web-worker thread in a browser so not to block the frontend UI at all
Take a look at demo/multithread/index.js and worker.js for example

[uzair004]

Also, best to process JSON in a web-worker thread in a browser so not to block the frontend UI at all
Take a look at demo/multithread/index.js and worker.js for example

it will solve processing problem but fetching huge data using fetch() or Ajax will still be a problem

NodeJS is single-threaded, but you can use a preinitialized pool of worker processes and
send each request to worker with currently lowest load

See demo/nodejs/node-multiproces.js and demo/nodejs/node-multiprocess-worker.js for example

I will look into it

[vladmandic]

Just FYI, JSON with 100 face descriptors is ~500KB and compressed below 200KB
And compression can be done on-the-fly by the web server, see server/serve.js for example

But if you have really large dataset with thousands of images, server could filter it down to rough matches of top 50 and then send small&compressed 100KB JSON to client for detailed processing

Actually, best to generate rough and detailed face descriptor when calculating it for each image to start with.

For example:

        // optional reduce feature vector complexity
        const reshape = tf.reshape(res, [64, 16]); // split 1024 vectors into 64 x 16
        const reduce = reshape.logSumExp(1); // reduce 2nd dimension by calculating logSumExp on it

This reduces complexity (and precision) of face descriptor from 1024-member-array to 64-member-array
Server can then run quick similarity search on small descriptor and pass best matches to client to run similarity search on full descriptor

It's just an idea...

[uzair004]

Server can then run quick similarity search on small descriptor and pass best matches to client to run similarity search on full descriptor

Nice idea, but i will need to process each image twice to get 64-member-array and 1024-memeber-array.

Should I save them in separate files and arrays ? If I did, searching for those top 50 matches in 1024-member-array will still kinda problem.

I can save 1024-member-array and 64-member-array in one file under same object. Further i will send only top 50 objects to client.

[xemle]

Hi. I am the maintainer of HomeGallery, a self hosted open source photo web gallery and would like to share my experience on that topic:

I extract similarity vector on the backend and do the similarity comparison on the frontend. I support image similarity AKA reverse image search and face similarity.

For the image similarity I did some investigation to reduce the data to fit into the browser with 87 bytes per image resulting a good balance between size and result. As example checkout the similarity of buildings on my demo gallery. 1.3 GB raw JSON data was reduced to 11 MB for 100,000 image set. The comparison (100,000 comparisons) takes less than 500ms on recent mobile devices.

For face similarity I am using face-api from @vladmandic (thanks again!). However I did not investigate into web optimization and the data is more or less original JSON, also the used version is quite old (0.13.3). Here the time is quite similar for a comparison.

Depending on your use case I guess it should be possible to get and process half million face similarity vectors into a recent browser. The similar vector should be browser optimized with a reduced data as @vladmandic proposed. If you have more than half million face vectors I suggest a backend solution.

[vladmandic]

Thanks for the comment.

Just to note, similarity matching algorithm is identical in both Human and FaceAPI. Difference is size of the descriptor - FaceAPI calculates vector which is 128-float-array while Human by default calculates 1024-float-array (which can be reduced to any size - but any reduction results in precision drop).

[uzair004]

As example checkout the similarity of buildings on my demo gallery. 1.3 GB raw JSON data was reduced to 11 MB for 100,000 image set

How did you reduce 1.3GB JSON to 11 MB ? Did you compress the images ? @xemle

[xemle]

I did not compress the image - I reduced the data of the image similiarity vector

• Similarity vector: infer layer from MobileNet v1 1.0
• Reduction for the browser (in base64): reduction algorithm The reduction was a try and error approach until I had a sufficient result. Mainly it is more important to have data points than the precision. At the end I reduce 1024 floats to 341 2-bit values with some limits, scaling and log2 mapping.
• Comparing in the browser (base64 decoded): image similar comparison function
• Similarity hash can be inspected by curl https://demo.home-gallery.org/api/database | jq .data[].similarityHash

I would try to find a good data reduction in the similar way for face vectors if time and need is available.

Does this answer your question?

[uzair004]

I went with Backend solution as of now, but i will look into details to try implement the way you did.

Thanks for your answer