-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run BirdNET weights in pytorch #29
Comments
Tried converting the tflite model to onnyx but the type of model was not supported. Getting the following error: "Unsupported TFLite OP: 89 REDUCE_MIN!" Not entirely sure what operators are exactly in tensorflow, but apparently onyxx only supports a certain number of them (and its lower than 89). I initially tried the tflite model as that what is used in the examples from BirdNet-Analyzer, but I am going to try other strategies, I am definitely going to be looking into MMdnn next. |
Can you try converting them into pytorch directly instead of onnx? Also, what was the full command that you ran? IIUC, there are both the tflite weights (optimized for cpu/embedded use), and there are the full model weights for gpu training. I'm not entirely familiar with tensorflow, but I think that might be the |
Yeah there are both tflite and .pb weights. I am going to try working with the .pb weights today. Onnx does not seem to have the ability to do direct transfer to pytorch. It seems you have to first transfer to onnx then to pytorch. I am going to look into MMdnn today. |
It looks like mmdnn has the ability to convert to any supported format as long as it can be represented in their intermediate format. I found this command on their README that converts a tensorflow checkpoint into a pytorch model: $ mmconvert -sf tensorflow -in imagenet_resnet_v2_152.ckpt.meta -iw imagenet_resnet_v2_152.ckpt --dstNodeName MMdnn_Output -df pytorch -om tf_resnet_to_pth.pth |
Some updates. MMDNN has not been updated to work on tensorflow 2.0 (current version it works with is 1.15) and so there is some incompatibility I think between the saved models in BirdNet Analyzer and MMDNN. In addition, I have been working on just loading the model in and looking at the architecture. |
Current issues with mmdnn are the following: mmconvert \
-sf tensorflow \
-iw vendor/BirdNET-Analyzer/checkpoints/V2.2/BirdNET_GLOBAL_3K_V2.2_Model/saved_model.pb \
--inNodeName INPUT \
--inputShape 144000 \
--dstNodeName CLASS_DENSE_1 \
-df pytorch \
-om tf2torch_saved_model gives the error "Error parsing message with type 'tensorflow.GraphDef'".
Running onnyx there are issues with certain operators used in tensorflow. This is pointed out on their github page: I am pretty sure (but not entirely convinced?) it is this giving the following errors when trying to convert both the tflite file and the .pb file: python -m tf2onnx.convert \
--saved-model ../vendor/BirdNET-Analyzer/checkpoints/V2.2/BirdNET_GLOBAL_3K_V2.2_Model/ \
--output data/logs/BirdNet0 \
--opset 17 gives the error:
whereas python -m tf2onnx.convert \
--opset 17 \
--tflite ../vendor/BirdNET-Analyzer/checkpoints/V2.2/BirdNET_GLOBAL_3K_V2.2_Model_FP32.tflite \
--output model.onnx gives the error
|
Just edited the commands for formatting. |
I took a gander at this -- it looks like we might want to build some of the tensorflow tools here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms#inspecting-graphs In particular, I'm interested in getting a dump of the graph and to take a look at some of the operations. RFFT2D presumably means the real part of the 2-dim fast fourier transform, but I'm not entirely sure what consumers are in the context. https://www.tensorflow.org/api_docs/python/tf/signal/rfft2d I took a look in this file for the error code: https://github.com/onnx/tensorflow-onnx/blob/ff139a41e0b34b0312f179ae66c2f41034ec2c72/tf2onnx/onnx_opset/signal.py#L116-L121 Not entirely sure what modifications are necessary in order to transform the graph so we can use it in onnx, but having the ability to inspect the graph is the first step. Some of those graph transformations ("freezing the graph") might be useful too. I think the crux of the solution will either be patching onnx or writing a graph transform of some kind so it's compatible with onnx. |
@zemm16 managed to get the model running on onnx via #51 See this notebook for details on how you can reproduce the results: https://github.com/dsgt-birdclef/birdclef-2023/blob/main/notebooks/acm-20230225-00-tensorflow-frozen-graph.ipynb It might be useful to start directly off with the onnx model, which can be found here: gs://birdclef-2023/data/models/birdnet-onnx-v1.onnx There are a couple of paths from here:
We should probably have a notebook that shows off the workflow using pytorch to generate an embedding, and to also label a section of audio. |
This is awesome. I will check out tomorrow. I was discussing onnx with some people the other day and was told that converting from onnx to pytorch is much easier than tf to onnx so might not have many issues. Fingers crossed. |
I need to review some of the links @acmiyaguchi you provided so I fully understand the model conversion works but it seems onnx2torch is not working because at some point the conversion file can't find nodes correctly in the onnx model. I reproduced the error in emm-20230226-00-onnx2pytorch.ipynb |
Linking #52 |
Another thing to consider -- maybe having the onnx model is enough, and we can consider moving birdnet inference up into the data loader layer (pre-processing) instead of treating it as part of the model? Trying to fit a 144k float32 vec into GPU memory is going to be memory intensive, so it might actually be a good idea to keep it on the CPU? |
Pursuing building the onnx model into data loader to extract embeddings |
We're probably going to settle down on pytorch in this project, just based on prior experience. However, two of the models that we want to use are written in tensorflow. These are BirdNET and Bird MixIT.
For BirdNET, the weights can be found under the
vendor
directory. Make sure that you checkout all the submodules:See this directory: https://github.com/kahst/BirdNET-Analyzer/tree/cb3707813f4465922823bcfba31358f6c5d0c370/checkpoints
We might be able to do this by exporting to onxx. I'd recommend using https://github.com/microsoft/MMdnn to transform the layers (I haven't used the tool myself).
I'd start off by taking a look at audio.py and model.py, and try to reproduce embeddings on an audio file using tensorflow directly. Afterwards, modify the workflow for pytorch after converting the weights.
The text was updated successfully, but these errors were encountered: