Bird-Model with index-error. How to resolve? #3

ozett · 2022-01-02T18:45:18Z

Great Happiness for the new dood2!!
hoped that it solved my problem with the bird-model.

https://tfhub.dev/google/lite-model/aiy/vision/classifier/birds_V1/3

as is spent the whole day to get it succesfull running in a bad python script - the new doods2 arrived.
great. 🌹

doods 2 loads the model and the label-file
(csv-format with comma, but also error without commas)
but throws an error on detections in the index

some googling said: convert the model to "newer" tf-lite ?

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     192.168.14.135:56005 - "GET / HTTP/1.1" 304 Not Modified
INFO:     192.168.14.135:56005 - "POST /image HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/opt/doods/api.py", line 38, in image
    detect_response = self.doods.detect(detect_request)
  File "/opt/doods/doods.py", line 108, in detect
    ret = detector.detect(image)
  File "/opt/doods/detectors/tflite.py", line 64, in detect
    classes = self.interpreter.get_tensor(self.output_details[1]['index'])[0] # Class index of detected objects
IndexError: list index out of range

both label-files dont work:

The text was updated successfully, but these errors were encountered:

ozett · 2022-01-02T19:02:47Z

i downloaded the tf-hub model,
than untarred it and saved to the model-dir with the label-file.
looked to me as tensorflow1.pb

to use the webgui of doods2 for image upload (great!)
(no selection of detector in the gui yet)
in the .yaml i named this one "default", but got an error.

any ideas?

ozett · 2022-01-02T19:09:02Z

loading like this works.

but as i am no coder an no AI specialist. the secret may lies in "sequential model"?
https://www.tensorflow.org/guide/keras/sequential_model

does doods2 support this?

snowzach · 2022-01-04T13:51:39Z

The main issue here is that you are using an image classification model vs what DOODS uses is an object detection model. Image classification looks at the whole image and then gives you it's estimation for each class that is matches that class. So you get a number for every value in your labels file that it's that type of bird. Presumably you pick the top one.

Object detection models take an image and then provide bounding boxes within an image of the class/label and a percentage.

The inputs and outputs are different shapes and thus doods doesn't know how to process the image classification models. I can look at adding support for it though. It doesn't look too terrible. There's a few other things I want to get done first though.

ozett · 2022-01-05T08:39:08Z

GREAT news!
would really appreciate to see this integrated as a detector,
so to say it right: OBJECT-Detector, not IMAGE (classifier)-Detector.

i assumed the difference of that models would have been handled the same, by code.
but seems not the case.

would be really great, if doods2 could be used with that object-model.

i will be patiently listening here for the next development step.... 👂

thx!! 💐

snowzach · 2022-01-06T05:03:02Z

Okay, I just made the change. It wasn't too bad. Essentially it will tag the entire image so if you want to filter by confidence, do not put anything in the regions section, just make a broad config. This will return anything with greater than 50% confidence. FWIW, the birds model seems to return numbers between 0-100 and not 0-1 like other models. So take the confidence you want and multiply by 100. (That's why I have 5000 to detect at 50%)

{
  "detector_name": "birds",
  "detect": {
    "*": 5000
  }
}

ozett · 2022-01-06T11:28:00Z

will try out instantly.. 😺

ozett · 2022-01-06T11:51:43Z

i have to pull a beta docker-image?
you used the config as in my first configuration.yaml ?

ozett · 2022-01-06T11:54:03Z

i try this:

ozett · 2022-01-06T11:57:50Z

no error - but where is the birds-name (classification).
no on the gui available? or am i doing something wrong?

ozett · 2022-01-06T12:02:39Z

seems somehow to work, no error.
but does the result dont show up on the gui?

by the way: million thanks for putting this in that fast! 🥇

https://upload.wikimedia.org/wikipedia/commons/thumb/2/29/ParusCaeruleus.jpg/300px-ParusCaeruleus.jpg
https://upload.wikimedia.org/wikipedia/commons/thumb/e/e1/Haussperling_092-2.jpg/300px-Haussperling_092-2.jpg

ozett · 2022-01-06T12:09:25Z

i thought the model will classify the bird-names.

maybe the return-value is not a detection-confidence,
instead the list-row of the bird-labels.csv, means the name? ~ 0-95...

snowzach · 2022-01-06T13:57:25Z

It was working, since it's a image classifier, the detection box was the entire picture and thus the label, usually being in the upper left corner, was out of the frame of view. I made a change that moves the label to the inside bottom left corner if it's too tall. I may just move the detection labels to the bottom left corner permanently. Oh, as a side note, I believe the detection range on that model goes from 0-255 and not 0-100 so use 0-25500 for your detection percentage.

ozett · 2022-01-06T18:10:04Z

somehow working with my bird,
not really readable...

would love to see the response-json on the gui,
as like the request...

sidenote: as i am not that used to docker i am really supprised that pulling the new image only was a millisecond.
assumably thats why docker pulled only litlle delta (changes). last time i thought it had not pulled anything new,
but that delta-pull is also really great!

ozett · 2022-01-06T18:54:12Z

after a second look,
it seems to me that your image-title was only the first-half of the birdname.

(the node-red response from doods on my detection was instead correct with the whole birdname)

snowzach · 2022-01-07T03:08:46Z

Make sure you are pulling the latest docker image. I added buttons that will show you the JSON now and it will sort the results so the best match is first. It's displaying the full name for me.

ozett · 2022-01-07T07:51:09Z

how do i check the latest version? on the gui?

i freshly pulled the doods2:latest (and will now try again)..

ozett · 2022-01-07T07:56:41Z

found new button, great!
thanks again!

ozett · 2022-01-07T08:22:38Z

the return-image on my node-red flow
looks like to have all detection results written on the image.
not only the highes match.
could this be different than the resulting image on the gui?

snowzach · 2022-01-07T13:01:20Z

It has all detections that match your confidence threshold. Try raising it to something like 5000. As long as you do pull the latest image like you did, you'll have the latest version.

ozett · 2022-01-07T14:16:46Z

anything more than 1 subtitel is hardly readable.
i would plege for only the highest confidence to put on the image.

even with threshhold 5000 the image mostly has round 34-78 subtitles ..
all written on top of each other....

ozett · 2022-01-09T09:52:31Z

sometime i get results with probablity-values,
but name classified as "unknown".

could this come from an index-error in handling the label-files.csv?
the first two entries in the original label-file arent birdnames...

the probabilty values seems correct (with map 0-255 to 0-100% values) and is above the threshold ....
maybe the doods2-response can include the index-nr ?

ozett · 2022-01-09T10:16:00Z

after small investigation:
maybe this is the result of the strange formatted labels.csv.
line 964 should be at the end, but its actually at place 2...

do you see a way to stay with that original-label file (for others to download the original from the hub,
but overcome this strange behaviour?

ozett · 2022-01-09T10:40:53Z

ugly:
in my experiment i added the second row (964,background) to the end (at 964),
where it should have been already normally in the file,
and than deleted the first two rows in that list.

that results in a list with 965 rows,
starting with row 0 and ending with 964 background.

so prediction in that special case is background, what sounds reasonable on that dark image,
and gives no index error .

still puzzled why the google-online-detections comes to a named result...
thats maybe not all to this bug... ??

ozett · 2022-01-09T11:03:34Z

hope you have an idea to the unknown..

ozett · 2022-01-09T11:49:40Z

testimage

ozett · 2022-01-09T16:27:40Z

some investigation:
seems that the provisioned label-file.csv is to short with only 964 lines,
should have been typical 1000 lines for a tensorflow-lite model for img-classification.

compare tensorflow-lite docu here: https://www.tensorflow.org/lite/examples/image_classification/overview?hl=en

https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_classifier?hl=en
https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/labels.txt

not experienced enough to read-out the models output-tensor...

snowzach · 2022-01-09T16:40:16Z

It really depends on how many labels the model is configured to detect. The issue is that the model is returning a number that is not in the labels file. I am working on pushing a new image.. hopefully today with PyTorch support and it will show you what number it is when it's unknown.

ozett · 2022-01-09T18:07:34Z

GREAT! really appreciate this,
i pulled a new image today, was a way bigger.
i expanded my virtual-host today, so i am prepared...

dont know if you can query the models metadata for possible detection-range? (0-964 or 0-1001)
this guy added english-names to the label-file, but ended also at 964
https://github.com/LaurentVeyssier/Bird_Classifier_Tensorflow_Colab_Notebook

maybe it needs version 3 of the model to fix a bug mentioned in the changelog?
"Fixes out-of-order logit to labelmap mapping in tflite metadata."
https://tfhub.dev/google/lite-model/aiy/vision/classifier/birds_V1/3

The issue is that the model is returning a number that is not in the labels file.

also with my tensorflow-code, but on the google gui-page with that model it works
correct with the test-images... suprisingly...

i will test with the new image... thanks a lot up to now..., ✋

ozett · 2022-01-09T18:25:57Z

stumbled on tensorflow-page about a model-inspector.
https://github.com/lutzroeder/netron

this all tells me nothing.
i only assume the model has only 1-965 output-values..

snowzach · 2022-01-09T19:10:59Z

I honestly don't know how to build the detectors. I just know you can define labels and detect on the labels be it 10 or 1000. I think 1000 is just the max.

ozett · 2022-01-09T19:25:32Z

i found on the tensorflow-pages, that the tflite-file can include the labels-files.
https://www.tensorflow.org/lite/convert/metadata?hl=en#read_the_metadata_from_models

and gotcha... label-file was included. and extracted if you rename .tflite to .zip
line 965 is "None". must be the limit

did you use the bird-label from the tflite-file,
or from the the .csv-file? seems nearly the same lines...

(the first 2 line from the strange .csv-file must be deleted
and the last line "none" added, than they are identical in length and content)

still wondering why the google-hub-gui has some other results than my code..

snowzach · 2022-01-09T20:51:44Z

Ahhh good detective work, I didn't realize they were bundled in. I might actually try to see if I can extract those.

ozett · 2022-01-10T07:53:06Z

now with pytorch/torch-hub support:
could this be possible to configure? (how to in the .yaml?)->

https://pytorch.org/hub/nicolalandro_ntsnet-cub200_ntsnet/

ozett · 2022-01-10T10:16:49Z

the .ipynb works...

how to configure this detector for the doods2.yaml ?
little bit unsure about this new pytorch modeltype...

ozett · 2022-01-10T10:21:22Z

wrong configuration?
or only 1x pytorch possible?

ozett · 2022-01-10T10:26:41Z

YES!
(would be nice if the download would be cached......)

ozett · 2022-01-10T10:32:35Z

not working..
for the pytorch-birdsmodell i will open up a new issue.. #19

snowzach · 2022-01-16T05:31:28Z

The metadata in tflite images doesn't seem to be a specific format. Since it's working otherwise, I'm closing this one.

ozett mentioned this issue Jan 2, 2022

Using custom tflite model snowzach/doods#35

Closed

ozett mentioned this issue Jan 6, 2022

[Support:] Working Example with Node-Red flow? #16

Closed

This was referenced Jan 10, 2022

[Support]: Pytorch model not working (birds) #19

Closed

Add support for PyTorch #8

Closed

snowzach closed this as completed Jan 16, 2022

Bird-Model with index-error. How to resolve? #3

Bird-Model with index-error. How to resolve? #3

Comments

ozett commented Jan 2, 2022

ozett commented Jan 2, 2022

ozett commented Jan 2, 2022 • edited

snowzach commented Jan 4, 2022

ozett commented Jan 5, 2022

snowzach commented Jan 6, 2022 • edited

ozett commented Jan 6, 2022

ozett commented Jan 6, 2022

ozett commented Jan 6, 2022

ozett commented Jan 6, 2022

ozett commented Jan 6, 2022

ozett commented Jan 6, 2022

snowzach commented Jan 6, 2022 • edited

ozett commented Jan 6, 2022 • edited

ozett commented Jan 6, 2022 • edited

snowzach commented Jan 7, 2022

ozett commented Jan 7, 2022

ozett commented Jan 7, 2022

ozett commented Jan 7, 2022

snowzach commented Jan 7, 2022 • edited

ozett commented Jan 7, 2022

ozett commented Jan 9, 2022 • edited

ozett commented Jan 9, 2022

ozett commented Jan 9, 2022

ozett commented Jan 9, 2022

ozett commented Jan 9, 2022 • edited

ozett commented Jan 9, 2022 • edited

snowzach commented Jan 9, 2022

ozett commented Jan 9, 2022 • edited

ozett commented Jan 9, 2022

snowzach commented Jan 9, 2022

ozett commented Jan 9, 2022 • edited

snowzach commented Jan 9, 2022

ozett commented Jan 10, 2022

ozett commented Jan 10, 2022

ozett commented Jan 10, 2022

ozett commented Jan 10, 2022 • edited

ozett commented Jan 10, 2022 • edited

snowzach commented Jan 16, 2022

ozett commented Jan 2, 2022 •

edited

snowzach commented Jan 6, 2022 •

edited

snowzach commented Jan 6, 2022 •

edited

ozett commented Jan 6, 2022 •

edited

ozett commented Jan 6, 2022 •

edited

snowzach commented Jan 7, 2022 •

edited

ozett commented Jan 9, 2022 •

edited

ozett commented Jan 9, 2022 •

edited

ozett commented Jan 9, 2022 •

edited

ozett commented Jan 9, 2022 •

edited

ozett commented Jan 9, 2022 •

edited

ozett commented Jan 10, 2022 •

edited

ozett commented Jan 10, 2022 •

edited