Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV 上傳的小包處理 #210

Open
ghost opened this issue Jul 18, 2019 · 0 comments
Open

CSV 上傳的小包處理 #210

ghost opened this issue Jul 18, 2019 · 0 comments
Labels
v2.0 tasks to be considered in V2

Comments

@ghost
Copy link

ghost commented Jul 18, 2019

先前上傳行程卡住的相機位置,雖然狀態 processing 沒有更新,但似乎仍有資料上去;這些相機位置的資料筆數均在 5000 至 30000 以上,在本機的測試發現,筆數多的時候,容易有報錯的情況。查了一下一般在 mongoose 和 mongodb 多筆數資料讀寫的能力限制,即使 db 本身理論上一次可以接受 16MB 資料的寫入,mongoose 仍不見得能順利處理該量級資料的上傳前處理,所以查到的建議是拆成 1000 筆左右的小包再由應用程式排程處理。我試著改寫上傳程式,將檔案拆成 1000 筆的小檔,再針對所屬的相機位置逐一上傳,目前在本機端沒有再看到 processing 狀態過了20分鐘還沒有更新為 success/failure 的情況。

此類的處理建議由應用程式以小包排程。

@ghost ghost added the v2.0 tasks to be considered in V2 label Jul 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v2.0 tasks to be considered in V2
Projects
None yet
Development

No branches or pull requests

0 participants