Make BulkInfoProviderService GREAT again.

[RawSmokeTerribilus]

Added logic to clean provider keys and handle corrupted keys during bulk processing.

Don´t hate me, vibe coded it. it works on my machine lol. Nah joking, it has been vibe coded but it does work perfectly fine in my container, i run the latest image. The feature is amazing! we were needing it working.

I have ONLY modified the file: src/Services/InfoProviderSystem/BulkInfoProviderService.php

Love you guys, amazing work!!!

[RawSmokeTerribilus]

Technical Review & Action Plan for Bulk Info Provider Fix

Subject: Fixing Invalid provider key type error in BulkInfoProviderService

1. Initial Problem Analysis

The primary reported symptom was that all bulk info provider jobs were failing immediately with a RuntimeException: No search results found for any of the selected parts.

Initial log analysis revealed that the root cause was not a lack of results, but a fatal error occurring before any API calls were made. The fieldMappings->providers array, which should contain simple string keys like "lcsc", was found to contain a corrupted data structure.

When a provider class (e.g., App\Services\InfoProviderSystem\Providers\LCSCProvider) was cast to an array, it included its private properties, resulting in a complex key structure contaminated with null bytes:

Corrupted Key Example:
\0App\Services\InfoProviderSystem\Providers\LCSCProvider\0lcscClient

The core logic was failing to match the expected clean key (e.g., "lcsc") against this corrupted key during in_array() checks, causing all providers to be silently skipped and leading to the "No search results" error.

The current working patch (the one submitted in this PR) implements a robust, multi-step cleaning and matching logic to handle this specific data corruption:

• Key Cleaning: New logic was added to performBulkSearch to iterate over provider keys. If a key is not a string, it's assumed to be the corrupted object/array.

• Null Byte Removal: The key is processed with str_replace("\0", '', ...) to remove the null bytes introduced by the private property casting.

• Greedy Regex Matching: The cleaned string (e.g., App\...LCSCProviderlcscClient) is processed with a greedy regex preg_match('/.*\\\\([\w]+Provider)/', ...) to extract the last valid class name (e.g., LCSCProvider), correctly ignoring InfoProvider from the namespace.

• Helper Function: A private helper method, providerKeyMatches(string $cleanedKey, array $providerList), was created. This function centralizes the new cleaning logic, allowing a clean key (like "lcsc") to be correctly matched against a list still containing the corrupted key.

• Patching Call-sites: All instances where in_array() or array_intersect() were used to check for a provider key (in processBatchProviders, processRegularProviders, and collectKeywordsForProvider) were replaced with calls to the new providerKeyMatches() helper.

Status: This patch successfully resolves the primary bug. The main bulk import workflow (Step 1) now correctly parses the provider keys, collects the part keywords, and successfully fetches results from the provider API.

3. Current Status & New Finding (The "Research All" Bug)

While Patch 11.1 fixes the main workflow, further testing revealed a minor edge case. When a job is already in Step 2 (review), clicking the "Research All" button (/job/{id}/research-all) fails.

Log analysis of this specific action shows that the performBulkSearch function is being called with a fieldMapping that contains an empty array ([]) as one of the providers.

The cleaning logic in Patch 11.1 was not designed to handle an empty array (it expects an object or a string). This causes it to log an Invalid provider key type AFTER cleanup {"providerKey":[],"type":"array"} error and fail the "Research All" process.

4. Action Plan (Final Fix)

The fix for this final edge case is minor and builds directly upon the working code in this Pull Request.

• Objective: Make the key cleaning logic in performBulkSearch robust enough to handle these "empty array" providers silently.

• The Fix: A single check will be added to the performBulkSearch loop. Before attempting to clean a non-string key, it will first check if it's an empty array.

Conceptual Code (to be added to performBulkSearch):

// Inside the main provider loop...
if (!is_string($providerKey)) {
    
    // [NEW FIX]
    // Silently skip if the provider mapping is just an empty array
    if (is_array($providerKey) && empty($providerKey)) {
        $this->logger->info('Skipping empty array provider key from mapping.');
        continue; 
    }
    // [END NEW FIX]

    // ... existing cleaning logic for corrupted objects ...
}

(Note: A previous attempt to implement this fix (Patch 11.2) contained a fatal syntax error ($this.logger vs $this->logger) which caused a 500 error. This was an authoring mistake and has been identified.)

Conclusion: This Pull Request (Patch 11.1) solves 99% of the critical bug. The final research-all issue is understood and a minor, safe addition (as shown above) will resolve it completely.

[Stephane-Peel]

Tested this patch today.

Seem to work fine thx for this !

Edit : Selecting a large amount of reference (80 parts) seem to hit the Mouser API limitation.
Maybe slow down the process ?

[RawSmokeTerribilus]

Tested this patch today.

Seem to work fine thx for this !

Edit : Selecting a large amount of reference (80 parts) seem to hit the Mouser API limitation. Maybe slow down the process ?

Hello! i am glad that you liked it! I'll take a look at Mouser's limit but, even slowing down the queries, part-db itself will break when launching big bulk import jobs. Even setting a bigger memory limit, jobs bigger than 100-150 queries will return an error 500 code. Anyways, as i would like to fix the research-all feature inside bulk-import, i'll do my best to also fix Mouser's queries timmings. For now, try to launch a limited amount of parts to update. It would be awesome if you share some logs about Mouser's limits, my parts come from LCSC so i haven't had any issue.

[jbtronics]

Tested this patch today.
Seem to work fine thx for this !
Edit : Selecting a large amount of reference (80 parts) seem to hit the Mouser API limitation. Maybe slow down the process ?

Hello! i am glad that you liked it! I'll take a look at Mouser's limit but, even slowing down the queries, part-db itself will break when launching big bulk import jobs. Even setting a bigger memory limit, jobs bigger than 100-150 queries will return an error 500 code. Anyways, as i would like to fix the research-all feature inside bulk-import, i'll do my best to also fix Mouser's queries timmings. For now, try to launch a limited amount of parts to update. It would be awesome if you share some logs about Mouser's limits, my parts come from LCSC so i haven't had any issue.

I think the correct approach for that would be to make large batch processing asynchronously from any web loading in some separate worker tasks. Symfony/messenger offers functionality for that quite easy but it is not really used anywwhere so far.

If that runs asynchronously then you can split up the part into smaller batches, as time doesn't matter.
With scheduler there are even way to run stuff periodically or planned, so you could use that to "circumvent" the api rate limits.

Just start the import and come back later.

[RawSmokeTerribilus]

Tested this patch today.
Seem to work fine thx for this !
Edit : Selecting a large amount of reference (80 parts) seem to hit the Mouser API limitation. Maybe slow down the process ?

Hello! i am glad that you liked it! I'll take a look at Mouser's limit but, even slowing down the queries, part-db itself will break when launching big bulk import jobs. Even setting a bigger memory limit, jobs bigger than 100-150 queries will return an error 500 code. Anyways, as i would like to fix the research-all feature inside bulk-import, i'll do my best to also fix Mouser's queries timmings. For now, try to launch a limited amount of parts to update. It would be awesome if you share some logs about Mouser's limits, my parts come from LCSC so i haven't had any issue.

I think the correct approach for that would be to make large batch processing asynchronously from any web loading in some separate worker tasks. Symfony/messenger offers functionality for that quite easy but it is not really used anywwhere so far.

If that runs asynchronously then you can split up the part into smaller batches, as time doesn't matter. With scheduler there are even way to run stuff periodically or planned, so you could use that to "circumvent" the api rate limits.

Just start the import and come back later.

Understood! Moving to an async worker/queue system sounds like the perfect long-term solution to avoid timeouts and rate limits without freezing the UI.

However, that architecture change is a bit above my current PHP skills (I'm mostly an electronics guy fixing what I broke!).

For now, this PR just fixes the crashing bug so the feature works again. Maybe we can open a separate 'Feature Request' issue to implement the Async/Symfony Messenger system later? I will definitely try to help, looks like a challenge!

[codecov]

Codecov Report

❌ Patch coverage is 42.00000% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.37%. Comparing base (95c5ab7) to head (e3a5061).
⚠️ Report is 34 commits behind head on master.

Files with missing lines	Patch %	Lines
...ces/InfoProviderSystem/BulkInfoProviderService.php	42.00%	29 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1110      +/-   ##
============================================
- Coverage     58.41%   58.37%   -0.05%     
- Complexity     7275     7293      +18     
============================================
  Files           579      579              
  Lines         23147    23192      +45     
============================================
+ Hits          13521    13538      +17     
- Misses         9626     9654      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jbtronics]

So I looked into this, and the underlying problem is that an object instance of an info provider was passed, where the provider key (a string) was expected. Unfortunatley PHP cannot ensure the types of an array at runtime and as it was filled by a form the static analysis couldn't catch it.

The AI proposed code does a lot of complicated to stuff to extract the provider key out of a stringified version of the whole object. That is not really elegant and will probably break quite easily.
The much easier and elegant approach is to just call ->getProviderKey() on the object, and it will return its provider key.

I have implemented this fix in the master branch now already, and it will be part of the next Part-DB release.

[RawSmokeTerribilus]

So I looked into this, and the underlying problem is that an object instance of an info provider was passed, where the provider key (a string) was expected. Unfortunatley PHP cannot ensure the types of an array at runtime and as it was filled by a form the static analysis couldn't catch it.

The AI proposed code does a lot of complicated to stuff to extract the provider key out of a stringified version of the whole object. That is not really elegant and will probably break quite easily. The much easier and elegant approach is to just call ->getProviderKey() on the object, and it will return its provider key.

I have implemented this fix in the master branch now already, and it will be part of the next Part-DB release.

Love you! AI is never the way. thanks for taking a look into it, for real, thanks!