Not completing discovery

[johnwhobbs]

I have multiple projects with hundreds of .stl files. All of the files are scanned and added to the library; however the process that creates the PNG files for each one just stops at some point and will never restart, even if I start a new discovery scan or reboot the container. And to accomplish this, I had to add .stl files to the library folder a little at a time. If they were all in there at first start of the Agent container, discovery would get to some point and just stop. Files aren't added to the library and PNG files are not created.

[EduardoOliveira]

Hello,
Could you please provide the logs of the agent container during the discovery phase?
Doing that on a fresh installation, meaning an empty data folder ( not library ).

Thank you

[johnwhobbs]

fd5c860a6fa2ecff885ddb2ea896be93b4bedbb954c780f1f2aee3f69b05e54a-json.log

[EduardoOliveira]

From the logs, you have a single project with a tone of files, right?
The project does show up in the UI right?
Are models missing from the UI?
Are some models present but without the thumbnail?
Can you post a screenshot?
In what hardware are you running the container? CPU RAM and architecture (arm or amd64)?

Sorry for all the questions but I need to understand the problem, from the logs, it discovers the content, looks like the assets are created, they are sent to the processing queue, but there are no errors....

[johnwhobbs]

There are 2 projects, one with 511 files and one with 656. Only one project shows in the UI and it does not contain all the models and most of the thumbnails are missing. I am running this on a QNAP NAS with quad core AMD64 CPU and 32GB RAM. I also tested on my personal PC which has an Intel 13900K and 64GB RAM. Same behavior.

[johnwhobbs]

When browsing the SQL Lite database, there are only 314 records in the project_assets table and 1 record in the projects table. This matches what is shown in the UI, but does not match what is in the library folder.

[johnwhobbs]

Any ideas?

[EduardoOliveira]

Hi, I've looked a bit at it, I seems there is a deadlock behaviour during the discovery process.
Given the most recent changes I think it's related to the fronted notifications....
I haven't had a chance to try, but some combination of starting mmp and going to the Web UI, refreshing to make sure the event channel is connected, avoiding the channel full error messages.

I need to confirm my theory and make a fix...

[johnwhobbs]

Thanks. I would be happy to test it when you are ready

[EduardoOliveira]

I've found the issue.
Currently, the render queue is limited to 256 pending requests and the processing queue limited to 100 requests...
If your library is BIG they fill up and enter a deadlock state...
As a quick and dirty fix, I'm gonna increase the size to 9999999, it should be enough...
I'm currently reprocessing a 17gb library, and it's working...
I'll post back when the bug fix is published

[EduardoOliveira]

@johnwhobbs Could you please test this package https://github.com/Maker-Management-Platform/agent/pkgs/container/agent/220710051?tag=130-not-completing-discovery ?

[johnwhobbs]

Works great now. All models present and thumbnails created for each. Thanks!!!!

[EduardoOliveira]

I'm gonna push this as an hotfix