docs: add disk usage / memory usage benchmark table

[ZiniuYu]

No description provided.

[codecov]

Codecov Report

Merging #751 (d2d7be2) into main (4d069a8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #751   +/-   ##
=======================================
  Coverage   81.78%   81.78%           
=======================================
  Files          17       17           
  Lines        1208     1208           
=======================================
  Hits          988      988           
  Misses        220      220           

Flag	Coverage Δ
cas	81.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96923f1...d2d7be2. Read the comment docs.

[hanxiao]

what about VRAM? thats very important.

[ZiniuYu]

This is in fact the VRAM. Maybe change the title. Do we need both RAM and VRAM statistics?

[hanxiao]

need a column on VRAM

[hanxiao]

on RAM and vRAM, either the number is wrong or the column names are wrong. Please check again

[ZiniuYu]

I quickly reran the test and noticed that the RAM usage is under-calculated by ~20%. The table is updated

[hanxiao]

This number is impossible. In general, I believe the whole column is wrong.

• We run 2 ViT-L/14-336px on Berlin GPU for the last 2 months and together they take 5GB VRAM.
• see also similar usage on Google Colab https://jina-ai.slack.com/archives/CTH1CMP8V/p1655126173106729

[ZiniuYu]

In fact, the VRAM data is not wrong. I think the reason I got a large VRAM value is that I was using a mini-batch of size 256 in clip-server. If I stick to the default value, the VRAM indeed drops to ~4GB.

[ZiniuYu]

I think I should clarify on that in the docs

[hanxiao]

No, you should clarify it in the benchmark docs.

You should use the default one for benchmark otherwise it is meaningless.

• Imagine the reader install clip-as-service and start to use it in the first time. Your table do not explain how much VRAM they would expect when running it out-of-the-box. People read it, "oh, 11GB that much? oh it uses batch_size 256, then what is the default batch_size and what is the default cost?" see? your benchmark does not solve the first question that users ask. In fact, it raises two questions for the readers: what is the default_batch and how much does it cost by default?
• Think you buy oranges in the grocery store. The price is always measured on the basic unit. One orange 1 usd, then people can derive a dozen of oranges is probably at 12$ or so. You don't often see that the shop only tells you the price of a dozen, and you have to infer it by yourself. which is basically what you are doing right now. You tell the reader the cost of batch_size 512, and user have to infer the vram cost of batch_size 8 (default, aka "the basic unit")

[ZiniuYu]

I see. Using a different batch size indeed introduces some confusion. Will update the table

[hanxiao]

• please fix as my comment
• please resolve conflict

[numb3r3]

@ZiniuYu please fix the commit message issue.

Error: You have commit messages with errors
⧗   input: docs: RAM usage
✖   subject must not be sentence-case, start-case, pascal-case, upper-case [subject-case]
✖   found 1 problems, 0 warnings

[github-actions]

📝 Docs are deployed on https://ft-docs_benchmark_table--jina-docs.netlify.app 🎉

[hanxiao]

looks like a test failing

[ZiniuYu]

looks like a test failing

That's weird. Fixed by reran the failed one. Anyway, learned a lot of lessons from this task🥹