Labels: Load and use a custom TensorFlow model

[Kazakuri]

It'd be nice to have some sort of documented procedure for loading a different TensorFlow model into PhotoPrism.

I don't have much experience with machine learning so I may be missing the mark with some of my assumptions but I tried to look into it a bit and wrote down my findings here.

---

From my initial investigation it looks like NASNet is downloaded when building the Docker image, and then the model is loaded from a static location in the application config.

By default that's ~/.local/share/photoprism/resources/nasnet so to replace the TensorFlow model you can just mount that folder with a new asset or otherwise change the assets-path in the config file.

From there I start to get a bit lost, when loading the model it seems like it loads a graph that's tagged with the name photoprism and labels from labels.txt in the same folder. It might be helpful to make the name of the tag that's loaded part of the configuration to make changing out models easier.

The documentation I found for LoadSavedModel doesn't tell me much, but my assumption is that it's only capable of loading protocol buffer models, so models must be trained and saved as .pb files or somehow converted to a .pb file.

The other thing I've noticed when looking through the code is that PhotoPrism uses models trained at 224x224, so I assume that if a custom model was trained at a different size then those numbers would need to be changed as well.

---

As a summary of what I've gathered it seems like these are the steps required:

1. Get your model into the correct format (.pb).
   • I'm not sure how possible this is, but maybe provide resources on how to do this for other common model types? Ex. h5, cntk, pth/pkl
2. Update the name of the graph being loaded from the model.
   • Either by updating the source code, or ideally a config file change.
3. Update the resolution that the images are scaled to.
   • Either by updating the source code, or ideally a config file change.
4. Mount your model folder as a volume to ~/.local/share/photoprism/resources/nasnet.
   • This should contain saved_model.pb and labels.txt.

---

IssueHunt Summary

Backers (Total: $60.00)

• $60.00 have been anonymously funded.

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[lastzero]

You are right, it's not that easy to change the model. We might build support for multiple models later. One of the reasons is, that models are trained using different resolutions, so you can't just use a different model as you need thumbs that have the right size and you also need to know how to interpret the labels (linguistics). On top, you need the model in the right format for TensorFlow for C. I had to write some Python for that. Not easy to automate.

[JimHinson]

Would it work to convert all images and thumbnails from the first set, then use the model on the second set? Thanks! From: Michael Mayer <notifications@github.com> Sent: Wednesday, July 24, 2019 1:12 AM To: photoprism/photoprism <photoprism@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [photoprism/photoprism] Loading a Custom TensorFlow Model (#127) You are right, it's not that easy to change the model. We might build support for multiple models later. One of the reasons is, that models are trained using different resolutions, so you can't just use a different model as you need thumbs that have the right size and you also need to know how to interpret the labels (linguistics). On top, you need the model in the right format for TensorFlow for C. I had to write some Python for that. Not easy to automate. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fphotoprism%2Fphotoprism%2Fissues%2F127%3Femail_source%3Dnotifications%26email_token%3DAC4XNEOZUAUE2H62YFGS33TQA7QAHA5CNFSM4IGLGT5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VF7CY%23issuecomment-514482059&data=02%7C01%7C%7C1a5b8790e3e04fb2bac908d70ff5647e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636995418928099088&sdata=wInfphUYgRKbEtPhYBxe6%2BjqNOX3V%2FfxmCRfWkmHiZM%3D&reserved=0>, or mute the thread<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAC4XNEJQBYA7UN3ZSW4ESRLQA7QAHANCNFSM4IGLGT5A&data=02%7C01%7C%7C1a5b8790e3e04fb2bac908d70ff5647e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636995418928109093&sdata=S3FYasiE2%2FQ3vc0KAGEewOmqOeiT1Prlfm%2BI64nOwB4%3D&reserved=0>.

[JimHinson]

Sorry, let me clarify: convert all images and thumbnails to the same resolution on both sets, then use the model from the first on the second set? From: Jim Hinson Sent: Wednesday, July 24, 2019 8:37 AM To: photoprism/photoprism <reply@reply.github.com>; photoprism/photoprism <photoprism@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: RE: [photoprism/photoprism] Loading a Custom TensorFlow Model (#127) Would it work to convert all images and thumbnails from the first set, then use the model on the second set? Thanks! From: Michael Mayer <notifications@github.com<mailto:notifications@github.com>> Sent: Wednesday, July 24, 2019 1:12 AM To: photoprism/photoprism <photoprism@noreply.github.com<mailto:photoprism@noreply.github.com>> Cc: Subscribed <subscribed@noreply.github.com<mailto:subscribed@noreply.github.com>> Subject: Re: [photoprism/photoprism] Loading a Custom TensorFlow Model (#127) You are right, it's not that easy to change the model. We might build support for multiple models later. One of the reasons is, that models are trained using different resolutions, so you can't just use a different model as you need thumbs that have the right size and you also need to know how to interpret the labels (linguistics). On top, you need the model in the right format for TensorFlow for C. I had to write some Python for that. Not easy to automate. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fphotoprism%2Fphotoprism%2Fissues%2F127%3Femail_source%3Dnotifications%26email_token%3DAC4XNEOZUAUE2H62YFGS33TQA7QAHA5CNFSM4IGLGT5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VF7CY%23issuecomment-514482059&data=02%7C01%7C%7C1a5b8790e3e04fb2bac908d70ff5647e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636995418928099088&sdata=wInfphUYgRKbEtPhYBxe6%2BjqNOX3V%2FfxmCRfWkmHiZM%3D&reserved=0>, or mute the thread<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAC4XNEJQBYA7UN3ZSW4ESRLQA7QAHANCNFSM4IGLGT5A&data=02%7C01%7C%7C1a5b8790e3e04fb2bac908d70ff5647e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636995418928109093&sdata=S3FYasiE2%2FQ3vc0KAGEewOmqOeiT1Prlfm%2BI64nOwB4%3D&reserved=0>.

[Kazakuri]

It looks to me like the makeTensor function is capable of resizing the thumbnails again after they're created.

The classifyImage function in the indexer will always pass in the tile_224, left_224, or right_224 thumbnails, but the call to imaging.Fill scales and crops the image to the dimensions you give it.

There might be issues with having that scale to a resolution bigger than the thumbnails, but if that function knew the resolution the model was trained at then couldn't it just handle the resizing there when classifying the image?

[lastzero]

We can also create thumbs on-demand, it's just slower and you need to configure out. Right now, the code is "hard wired" to only work with our standard model, but it should be relatively easy to add config options later.

[Kazakuri]

Small info dump update since I had a bit more time to play around with this today. I was able to convert some models fairly successfully using MMdnn.

An example of one of the models:

$ saved_model_cli show --all --dir tf

MetaGraphDef with tag-set: 'train' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 299, 299, 3)
        name: Input3:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 6681, 1)
        name: StableSigmoid786596:0
  Method name is: tensorflow/serving/predict

From that output, I changed the tag-set to train instead of photoprism, changed the graph operations to Input3 and StableSigmoid786596 instead of input_1 and predictions/Softmax, and changed the imaging.Fill width and height to 299 instead of 224.

After that I updated the labels.txt and gave it a shot but unfortunately, since the output type of the model is [][][]float32 instead of just [][]float32, the bestLabels function doesn't know how to process it. The inner array only ever seems to be a single element, but I don't know if there's an easy way to get around that limitation in the model.

For what it's worth though, after some minor modifications to bestLabels to accept a [][][]float32 and dump the arrays, it did look like I was getting back some pretty good results.

[lastzero]

@Kazakuri Shall we keep this issue open? What exactly should be the outcome / acceptance criteria?

[Kazakuri]

I guess it sort of depends on what you and the other PhotoPrism devs want to do about custom / multiple models, but things like the changing model graph name and the thumbnail resolution could probably be moved to into config options so that you're not forced to rebuild the code to swap out the model.

Like you said in your initial comment, there are still a handful of complications around getting a pretrained model into the right format that PhotoPrism is expecting, so if it's possible then some documentation in the wiki as a starting point for developers to go off of would be great. It could even end up being a whole other repository that's dedicated to doing model conversions into PhotoPrism's format but that's extremely complex and well outside the scope of PhotoPrism.

[lastzero]

We need to be careful not to solve problems that not (yet) exist. When adding a new model recently (NSFW detector), I figured the code needed is so different that abstraction would only complicate things. Results even depend on how you resize images and create tensors from them. See #175.

You are right that our docs could be improved and we also want to provide workshops for people who want to learn Go and/or TensorFlow. Maybe we can do this next year when our community becomes more active. Even in Berlin, days have only 24 hours and we - the core team - also need to sleep and eat ;)

Are you interested in contributing?

[Kazakuri]

Ah, that makes sense, yeah. When I was making this issue I did think of NSFW detection, but I was initially looking more towards models for hair color and gender classification or a completely separate use case for classifying digital art using something like Danbooru2018.

I'm by no means a machine learning / TensorFlow expert, but it is something I'm fairly interested in getting more experience with. Similarly, I know enough about Go to hack around a project that uses it, but I wouldn't say I can program well using it.

I haven't had much of a chance to tinker with this project since September but it is still something I'm looking at getting back to playing with, so I would be happy to contribute whatever I can.

[issuehunt-oss]

An anonymous user has funded $60.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.