ArticleSnapshot table rotation

[pm5]

ArticleSnapshot 用多個 table 去 rotate，確定 parse 完的 table 就 dump 出來備份，再直接砍掉，這樣會比較好管理。

• 用 article ID 切換 table
  • discover 步驟：
    1. 寫進新的 article，取回 article id
    2. 根據 article id 取得 snapshot table name
    3. 把 snapshot 寫進指定的 snapshot table
  • update 步驟：
    1. 取得要 update 的 article
    2. 根據 article id 取得 snapshot table name
    3. 把 snapshot 寫進指定的 snapshot table
• 先把大約一年份的 snapshot table 都開好？
• 要檢查哪些 table 可以砍掉

[pm5]

Wrote a tentative scheme for table rotation over here. Partitioning is nice because it is completely done by the storage layer. The crawler pipeline does not have to know about it so there would be fewer code changes.

[pm5]

Going to implement this by the end of March.

[pm5]

Since we do not have a mechanism yet to decide when can we truncate old snapshots, we will do it manually for the moment. This closes this issue.