Issue with Data Persistence and "Document not available" error after Server Migrationiscussion

[sharpdima]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

Hello Dify Team,

I have recently migrated my self-hosted Dify instance (version 1.11.2) to a new server. I managed to restore the PostgreSQL database, so all my Knowledge bases, document lists, and configurations are visible in the dashboard.

However, I am facing two critical issues:

Document Not Available: When I try to access or re-process existing documents in the Knowledge base, I get a "Document is not available" error. I have verified that the physical files exist on the server under the volumes/app/storage/upload_files/ directory, but they are nested inside a UUID sub-folder (e.g., upload_files/9a9fc24b.../0f47...docx).

Vector Data Loss: Since the Weaviate (vector database) was re-initialized during the build, the "Retrieval Test" and "Chatbot" cannot find any answers from the old documents, even though they appear as "Available" in the list.

My Goal: I want to re-index/restore these documents without re-uploading them manually, as the database entries and the physical files are both present on the server.

Questions:

What is the expected directory structure for upload_files in version 1.11.2? Should I move files out of the UUID sub-folder?

Is there a CLI command or a specific API call to trigger a "bulk re-sync" or "re-index" that forces Dify to read the existing physical files from storage and re-populate the vector database?

Current Setup:

Dify Version: 1.11.2 (Docker Compose)

Database: PostgreSQL

Vector Store: Weaviate

Thank you for your support.

[dosubot[bot]]

The UUID subfolder structure under upload_files is expected—Dify stores files using a storage key that includes these subfolders, and the database references this path. Do not move files out of their UUID subfolders; doing so will break the link between the database and the physical files, causing "Document not available" errors even if the files exist on disk. If you’re seeing this error, double-check that the storage_key in your database matches the actual file path on disk, including the UUID folders. Source

There is no built-in CLI or API command in Dify v1.11.2 to bulk re-index or re-sync documents from storage. After re-initializing Weaviate, vector data is lost and must be restored manually. Options include re-uploading documents, scripting an export/import, or (sometimes) changing the embedding model of the knowledge base to trigger an automatic index rebuild. For large knowledge bases, scripting is recommended, as there is no automated migration script for this process. Source

Always back up your database and files before making manual changes.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}