Reduce the memory usage of PDF asset file content parsing on the mobile

[88250]

Otherwise, it is easy to cause the application to crash when indexing asset files on the mobile.

[88250]

@nekrondev Hello, I tested a PDF here. The memory occupied by a single core is about 1G. If it is 4 cores, it will take more than 3G, so it will cause the mobile app to crash.

Do you currently have a better way to reduce memory usage? If you need the PDF sample, please send me an email (845765@qq.com), and I will send you this PDF sample as an email attachment, thank you.

[zxhd863943427]

2.10.2的移动端似乎并不能搜索资源文件。

[88250]

观察一下桌面端重建资源文件索引时的内存占用。

…

---Original--- From: ***@***.***> Date: Thu, Aug 31, 2023 22:06 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [siyuan-note/siyuan] Reduce the memory usage of PDF asset filecontent parsing on the mobile (Issue #9079) 2.10.2的移动端似乎并不能搜索资源文件。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.Message ID: ***@***.***>

[nekrondev]

Hello @88250 I'll have a closer look into this. The go-pdfium project writes about memory consumption if multiple workers are added. I think I'll do some investigation into dynamically allocating a worker pool if there is no memory pressure, i.e. read the available system memory and launch more or less workers.

[88250]

I'm not sure if there is a way to get the currently available memory on the mobile. Now the dev branch has limited the mobile to 1 core.

[nekrondev]

That looks like a good solution to me! I'll continue my tests with dynamic worker pool and see if I can provide some solution eventually. The problem could be that once allocated memory by the WASM runtime will only GC'ed when the go-pdfium pool will close releasing the runtime and allowing GC to cleanup heap and stack.

[88250]

@nekrondev

It was only found through today's test that the mobile does not support PDF parsing and an error will be reported under Android:

convert [/storage/emulated/0/Android/data/org.b3log.siyuan/files/siyuan/temp/convert/asset_content/u6uhcrz.pdf] failed: [could not instantiate webassembly module: module[] function[_initialize] failed : permission denied (recovered by wazero)

It will directly cause a crash under iOS (I guess wazero is not compatible with iOS).

For PDF parsing, let's give up mobile support for now, I describe this limitation in the user guide.