Pytorch->Deepstack detector not working with recently trained models

[Idefix0496]

Hi,
Sorry to annoy you again but I think yolov changed something in their stuff again because I'm no longer able to run doods2 on recently trained models.
I already tried pulling the latest Image from Doods2 while also creating a new Volume but to no avail.
That's the Error I'm getting while doods2 ist starting with a new model:

sudo docker run -it -p 8080:8080 --restart unless-stopped -v Test:/opt/doods snowzach/doods2:latest 2022-12-27 21:23:55,911 - doods.doods - ERROR - Could not create detector deepstack/pytorch: Can't get attribute '_rebuild_parameter_v2' on <module 'torch._utils' from '/usr/local/lib/python3.8/dist-packages/torch/_utils.py'> 2022-12-27 21:23:56,050 - uvicorn.error - INFO - Started server process [1] 2022-12-27 21:23:56,051 - uvicorn.error - INFO - Waiting for application startup. 2022-12-27 21:23:56,052 - uvicorn.error - INFO - Application startup complete. 2022-12-27 21:23:56,054 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

Putting the predecessor model back the Error dissapears.
Following I've also attached the command line Output of doods starting for the first time after pulling the Image:

Status: Downloaded newer image for snowzach/doods2:latest 2022-12-27 21:39:37,548 - doods.doods - INFO - Registered detector type:tflite name:default 2022-12-27 21:39:44,975 - doods.doods - INFO - Registered detector type:tensorflow name:tensorflow /usr/local/lib/python3.8/dist-packages/torch/hub.py:267: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour warnings.warn( Downloading: "https://github.com/ultralytics/yolov5/zipball/master" to /root/.cache/torch/hub/master.zip requirements: YOLOv5 requirements "gitpython" "scipy>=1.4.1" not found, attempting AutoUpdate... WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: You are using pip version 21.3.1; however, version 22.3.1 is available. You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command. Collecting gitpython Downloading GitPython-3.1.29-py3-none-any.whl (182 kB) Collecting scipy>=1.4.1 Downloading scipy-1.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB) Collecting gitdb<5,>=4.0.1 Downloading gitdb-4.0.10-py3-none-any.whl (62 kB) Requirement already satisfied: numpy<1.26.0,>=1.18.5 in /usr/local/lib/python3.8/dist-packages (from scipy>=1.4.1) (1.21.5) Collecting smmap<6,>=3.0.1 Downloading smmap-5.0.0-py3-none-any.whl (24 kB) Installing collected packages: smmap, gitdb, scipy, gitpython Successfully installed gitdb-4.0.10 gitpython-3.1.29 scipy-1.9.3 smmap-5.0.0

requirements: 2 packages updated per /root/.cache/torch/hub/ultralytics_yolov5_master/requirements.txt requirements: ⚠️ Restart runtime or rerun command for updates to take effect

YOLOv5 🚀 2022-12-27 Python-3.8.10 torch-1.13.0+cu117 CPU

Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.pt to yolov5s.pt... 100%|██████████████████████████████████████| 14.1M/14.1M [00:00<00:00, 30.2MB/s]

Fusing layers... YOLOv5s summary: 213 layers, 7225885 parameters, 0 gradients Adding AutoShape... 2022-12-27 21:40:19,227 - doods.doods - INFO - Registered detector type:pytorch name:pytorch 2022-12-27 21:40:19,367 - uvicorn.error - INFO - Started server process [1] 2022-12-27 21:40:19,368 - uvicorn.error - INFO - Waiting for application startup. 2022-12-27 21:40:19,369 - uvicorn.error - INFO - Application startup complete. 2022-12-27 21:40:19,370 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

As before for reference I'm attaching a freshly trained Coco128 Model which is also not working:

best_coco128_yolov5s.zip

Would appreciate it if you could take a look again :-)

[snowzach]

It seems like it might be a pytorch version issue. Do you know which version you trained it with?

[Idefix0496]

Yes it's this Version:

YOLOv5 🚀 v7.0-49-g3c1afd9 Python-3.8.10 torch-1.14.0a0+410ce96 CUDA:0 (NVIDIA GeForce GTX 1080 Ti, 11264MiB)

[Idefix0496]

Hey was this the Information you needed ?

[snowzach]

Pytorch 1.14 isn't released yet from what I can see.. Not sure I want to build with a pre-release.

[snowzach]

Hmm, okay, I tried with pytorch 2 (which is what 1.14 will be) and it still seems to be getting that error...

[snowzach]

Yeah, I can't find any version of pytorch that has the function _rebuild_parameter_v2 in it.. I wonder if this is some special fork.

[Idefix0496]

Hey so it seems that Models which are trained with at least Version:

YOLOv5 🚀 v7.0-71-gc442a2e Python-3.8.10 torch-1.13.1+cu117 CUDA:0 (NVIDIA GeForce GTX 1080 Ti, 11264MiB)

are working again.
I've got one in use since a couple of days without any issues.
I haven't touched the doods2 Container just trained a new Model with a new Image from yolov5.

I think they changed the pytorch integration back to 1.13.1. Nevertheless thank you for your effort's so far !

[snowzach]

I love problems that fix themselves.. :-)