Import Timeout and Missing/Transient Notifications for Large Spreadsheet Imports

[Shad0wTeam]

First Check

• This is not a feature request
• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the documentation, with the integrated search.
• I already read the docs and didn't find an answer.
• I can replicate the issue inside the Demo install.

Homebox Version

0.17.2

What is the issue you are experiencing?

Description:

I encountered an issue in the Homebox App when importing data from spreadsheets that contain over 100 rows. The problem occurs when attempting to import large datasets, causing the import process to time out. Below are the details of the issue:

Expected Behavior:

1. When importing smaller sheets, users receive a notification that quickly appears and disappears, confirming whether the import was successful or failed.
2. For large imports (over 100 rows), the user should still receive a persistent notification indicating whether the import was successful or failed. The notice should stay until the user interacts with it, rather than disappearing too quickly.

Actual Behavior:

1. When importing spreadsheets with more than 100 rows, the notification that confirms whether the import was successful or failed does not appear or stay visible long enough to be useful.
2. In fact, after the import completes, the notification never appears, and there is no clear indication of whether the import was successful or not.
3. To determine if the import was successful, I have to rely on watching the machine’s CPU usage, which only drops to normal levels when the import completes. This is a poor signal and adds unnecessary complexity to the process.
4. Additionally, the import dialog box (for large imports) stays open indefinitely and does not disappear once the import times out. This prevents the user from knowing the process has concluded.
5. The lack of a persistent notification makes it difficult to determine the import status. In cases where the notification is transient, I observed that it was very hard to spot, and I had to monitor the CPU usage for over an hour before seeing any indication of completion.
6. Even after waiting for a prolonged period (such as 1 hour for 300 records), the notification never appeared at all. The only way I could confirm the import was successful was by checking the last row of the spreadsheet and comparing it with the most recent record in the tool.

Suggested Fixes:

1. Persistent Notifications: Keep the import success or failure notification visible until the user clicks "OK" or interacts with it. This would provide better feedback to the user.
2. Timeout Handling: Ensure that if the import process times out, the import dialog disappears after a certain time, and an appropriate error message is displayed, indicating that the import did not complete successfully.
3. Clear Signals for Completion: Implement a clearer visual indicator (e.g., a progress bar or consistent notification) for larger imports so users don't have to rely on CPU usage as a signal for completion.
4. Consistent Dialog Behavior: The import dialog should automatically close after a successful or failed import, and the appropriate success/failure notification should be displayed to the user.

Additional Information:

• I observed the issue specifically with larger imports with lots of details in both description and notes columns, so smaller files (under 100 rows) seem to work as expected.
• The issue seems to be related to handling long-running imports that take too much time, causing the UI elements to behave unpredictably.

How can the maintainer reproduce the issue?

Steps to Reproduce:

1. Import a spreadsheet with over 100 rows into the Homebox App.
2. Observe that the import notification does not appear or stays too briefly to notice.
3. The import dialog remains open even if the process times out, with no clear indication that the import has finished.
4. Watch the CPU usage drop (if the machine is actively used during the process) to detect when the import has completed.

Deployment

Other, Docker (Linux)

OS Architechture

x86_64 (AMD, Intel)

Deployment Details

Running a Proxmox LXC with Homebox Installed

[Shad0wTeam]

Additional Note:

For clarification, in my most recent tests, the HB.import_ref field was populated for all records. However, initially, this field was not populated, which resulted in duplicates for each test, as would be expected. Despite the changes made to populate this field in later tests, the issue with the large sheet import still persists as described earlier.

[EdWorth120]

When you say "a spreadsheet with 100 rows" it was not clear what you mean, this is a CSV file with 100 lines? I tested a CSV file with 1315 items (lines), it finished in less than a minute. The system really have some problems, but the process finishes when the modal box closes and a toast message is briefly shown with the text "Import successful!" (for moreless 5 seconds) and disappears even if the user don't interact with it.

Independent of the size of the CSV file, the behaviour is the same, the modal box only closes when the process finishes, being it completed with or without errors and a toast message appears for moreless 5 seconds, showing if it was successful or if it failed.

There is another problem with actions that takes more than 10 seconds to finish and are not able to pass any kind of status (like how many items was imported) to the frontend show in the toast message, but the CSV import function doesn't use any response information from the API to show in the screen, so this problem is not applicable here.

I agree that there is no progress indicator, the modal boxes are not even using some sort of spinning gif to pass an impression that it is working, and the toast messages are fast, but I was not able to reproduce "messages not appearing for large files".

[tonyaellie]

Created a script to generate data:
https://github.com/tonyaellie/homebox-gen-data
Defintely seems to need work, love this clear message

[Shad0wTeam]

When you say "a spreadsheet with 100 rows" it was not clear what you mean, this is a CSV file with 100 lines? I tested a CSV file with 1315 items (lines), it finished in less than a minute. The system really have some problems, but the process finishes when the modal box closes and a toast message is briefly shown with the text "Import successful!" (for moreless 5 seconds) and disappears even if the user don't interact with it.

Independent of the size of the CSV file, the behaviour is the same, the modal box only closes when the process finishes, being it completed with or without errors and a toast message appears for moreless 5 seconds, showing if it was successful or if it failed.

There is another problem with actions that takes more than 10 seconds to finish and are not able to pass any kind of status (like how many items was imported) to the frontend show in the toast message, but the CSV import function doesn't use any response information from the API to show in the screen, so this problem is not applicable here.

I agree that there is no progress indicator, the modal boxes are not even using some sort of spinning gif to pass an impression that it is working, and the toast messages are fast, but I was not able to reproduce "messages not appearing for large files".

Sorry, been busy. Try again with say 50 additional custom columns and see the difference.

[Shad0wTeam]

Als, I created a repo to help folks wanting to use the API to manage their self hosted instance of homebox. But it seems the attachment uploader is now broken with the latest version of Homebox.

Here is the repo: https://github.com/Shad0wTeam/api-homebox-simple-poc-tools?tab=readme-ov-file

[tankerkiller125]

Als, I created a repo to help folks wanting to use the API to manage their self hosted instance of homebox. But it seems the attachment uploader is now broken with the latest version of Homebox.

Here is the repo: https://github.com/Shad0wTeam/api-homebox-simple-poc-tools?tab=readme-ov-file

We updated how attachments work in the latest release. Any code using it will need updates per our docs, or if you generated it from the open API scheme regenerated with the latest version.

[Shad0wTeam]

Als, I created a repo to help folks wanting to use the API to manage their self hosted instance of homebox. But it seems the attachment uploader is now broken with the latest version of Homebox.
Here is the repo: https://github.com/Shad0wTeam/api-homebox-simple-poc-tools?tab=readme-ov-file

We updated how attachments work in the latest release. Any code using it will need updates per our docs, or if you generated it from the open API scheme regenerated with the latest version.

Hmm, I wrote my own python library using the API documentation, i am guessing that documentation is not updated yet. Has the endpoint changed? The response I am getting from the endpoint does not align with the data outlined in the documentation. I am using this link: https://homebox.software/en/api/#/paths/v1-items-id--attachments/post

[Shad0wTeam]

Ok, filed a bug, as I struggled for a while to refactor to code to work on 0.19 before giving up. I now convinced there is a bug in the api for 0.19.

Here is my bug report: #778