AI: Allow configuration of custom classification models

[raystlin]

This commit adds the possibility of using custom models for classification via vision.yml configuration file.
The code will inspect the models and try to find their variable layout, inputs, number of outputs and so on, however, if the model returns logits instead of probabilities it must be specified on the configuration to add a softmax layer after the last layer of the model.

Notes

• NSFW and Faces can be configured and replaced, but the rest of the code is not really prepared for them.
• Please, take into account that download-nasnet.sh script has been modified to include labels21k.txt to nasnet path, so as to have a default implementation of 21k labels, although the rules for them have not been written yet.
• Also the test case internal/ai/classify/model_external_test.go has been set as optional via an environment var, as it is not an acceptance but one to perform on test photos against external models.

Related Issues

• Labels: Load and use a custom TensorFlow model #127

[lastzero]

It probably makes sense to comment this out (or remove it) until 21k labels are actually supported? As I mentioned in my last issue comment, I believe it would be easier to generate CLIP embeddings and implement a search with those than to generate 21,000 different labels suitable only for regular text-based searches?

[raystlin]

Yes, you are right. I modified the script so as to have some kind of classification (although the rules were not being applied), but if we are thinking of alternative ways of doing the ruling, it makes more sense to remove this.

[lastzero]

@raystlin Thanks a lot! That's awesome :)

After taking a (very) quick look at your changes, I have a few questions regarding their intended scope and functionality:

• It seems like the newly added PhotoInput abstracts the input thumbnail configuration, right?
• When you pass an image: Are there any constraints regarding size or formats?
• Should it automatically resize the image to match the configured dimensions?
• If the input image is not square or does not match the configured size or aspect ratio, can/should multiple crops automatically be created to cover the entire area?

Also, I'd like to ask the following just to be sure:

• When you run make test-go in a terminal, do all the existing and new unit tests pass? Have you observed any instabilities or performance issues?
• How is the test coverage, especially with regard to the new functionality? Should more tests be added before or after these changes are merged?
• Did you run make fmt-go to reformat the source files as needed?

[lastzero]

Having a vision.yml configuration example with your changes applied could help with testing. This would clarify what it should look like and enable us to easily compare it with the existing format and available options.

[raystlin]

Of course. I missed this one. This is the example I used for the transformer one, but I have to change it on the PR.

---
Models:
- Type: labels
  Name: transformer
  Version: Mobile
  Resolution: 224
  Meta:
    Input:
        Interval:
            Start: -1.0
            End: 1.0
    Output:
      OutputsLogits: true
- Type: nsfw
  Name: Nsfw
  Resolution: 224
- Type: face
  Name: FaceNet
  Resolution: 160
- Type: caption
  Resolution: 224
  Service:
    Uri: http://photoprism-vision:5000/api/v1/vision/caption
    FileScheme: https
    RequestFormat: url
    ResponseFormat: vision
Thresholds:
  Confidence: 10

[raystlin]

• PhotoInput abstracts the entry needed by the model: the input layer name, the expected height and width, the expected interval for input values ([0,1] or [-1,1] or whatever), the index of the output and so on. I try to deduce as much as I can from the model deserialization, but those assumptions can be overridden via vision.yml. Please, also notice that I left resolution value at Models level, this is because I assumed by the comment Labels: Load and use a custom TensorFlow model #127 (comment) that this parameter was going to be used to generate thumbnails in the exact size needed by the model. I think I misunderstood it, but we can always do a transformation like the one done on NSFW to solve it.
• Supported file formats are the same ones previously supported: jpg and png (I did not change the image decoding part of the code).
• Constraints are not enforced. As I previously said, I thought the input was being enforced to its correct size during the thumbnail generation. If this assumption is wrong, I am afraid I have to work on that part.
• make test-go runs and passes the tests. I have not seen performance or stability issues.
• I have added the file internal/ai/classify/model_external_test.go to test/compare the new model against the test data used for nasnet. The test downloads the models (which is really slow), runs them and, sometimes, the labels generated for some tests are not the expected ones. Because of this, I added an enviroment variable as a condition to run them.
• I ran make fmt-go but I missed one file: internal/ai/classify/model_external_test.go. I will add it to the commit.

[lastzero]

@raystlin Sorry for not getting back to you sooner! I was traveling, but am now back in the office. Please let me know if you consider this stable enough for release. I will then merge your changes, perform final testing, and create a new preview build :)

[lastzero]

@raystlin All the unit tests are green, so that's great!

As a general note, the inline comments should end with a period because they are used to automatically generate API documentation (not a big issue, though, because I can easily update them):

For our end user and developer documentation, it would also be helpful to have more information on the newly added configuration options, as well as a list of the models that you have tested and found to work?

Thank you very much! 🤗

[lastzero]

@raystlin I've merged your changes with a few edits for further testing and documentation. Thank you so much for working on this! ❤️

It would be good to have some specific configuration and usage examples ready for end users before releasing these improvements in the stable version. @graciousgrey and @omerdduran are happy to help with this.

[raystlin]

Sure, I will do it this afternoon. I will also bear it in mind for future contributions.

[lastzero]

You can now test these changes with the updated preview build:

• https://docs.photoprism.app/release-notes/#development-preview
• https://docs.photoprism.app/getting-started/updates/#development-preview