Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A couple of questions on EROFS #6

Closed
donmark8 opened this issue May 29, 2022 · 6 comments
Closed

A couple of questions on EROFS #6

donmark8 opened this issue May 29, 2022 · 6 comments

Comments

@donmark8
Copy link

Hi Gao Xiang,

Good works on EROFS! I have a couple of questions, hope you can help to answer them.

  1. Comparing with lz4, lzma will make the size small, but decompression time will be longer, do you have any benchmark test results for lz4 and lzma in terms of size and performance.
  2. Do you plan to add lzma support for EROFS-Utils in buildroot?
  3. Do you plan to add zstd support for EROFS in kernel?

Thanks,
--Don

@hsiangkao
Copy link
Collaborator

  1. yeah I did, actually lzma and lz4 are used for different scenarios, lzma is still an experimental feature which is used for small embedded devices like routers or other non I/O sensitive scenarios, but lz4 is for high-performance scenarios.
  2. I plan to do, but it really relies on a new stable XZ-utils release, which is still WIP;
  3. I plan to do, but I don't want to add a plain algorithm without any optimization and adaption. It still need some time to evaluate and do some like inplace I/O optimization.

@donmark8
Copy link
Author

Thanks for the reply. I am glad to hear that you will work on zstd for kernel and add more supports in buildroot. Keep up the good works!

@firasuke
Copy link

Is this still being worked on? The only thing stopping me from adopting EROFS is that it lacks support for zstd (SquashFS is compressing much better due to having zstd as an option).

@hsiangkao
Copy link
Collaborator

hsiangkao commented Jan 26, 2024

Is this still being worked on? The only thing stopping me from adopting EROFS is that it lacks support for zstd (SquashFS is compressing much better due to having zstd as an option).

If you think zstd support is vitally important, I can only say it's still working in progress.
But I cannot give an accurate timetable for the optimial Zstd (Cc @Cyan4973).

@firasuke
Copy link

Is this still being worked on? The only thing stopping me from adopting EROFS is that it lacks support for zstd (SquashFS is compressing much better due to having zstd as an option).

If you think zstd support is vitally important, I can only say it's still working in progress. But I cannot give an accurate timetable for the optimial Zstd (Cc @Cyan4973).

I really think zstd support is important, and I appreciate that you are working on it. No need for a timetable, I am just checking if it'll be supported in the first place, or if it got dropped.

Thanks for your time and effort on this!

@hsiangkao
Copy link
Collaborator

Is this still being worked on? The only thing stopping me from adopting EROFS is that it lacks support for zstd (SquashFS is compressing much better due to having zstd as an option).

If you think zstd support is vitally important, I can only say it's still working in progress. But I cannot give an accurate timetable for the optimial Zstd (Cc @Cyan4973).

I really think zstd support is important, and I appreciate that you are working on it. No need for a timetable, I am just checking if it'll be supported in the first place, or if it got dropped.

It's still considered but I don't have an accurate finish time.

Thanks for your time and effort on this!

Thanks for your request too.

intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue May 8, 2024
Add Zstandard compression as the 4th supported algorithm since it
becomes more popular now and some end users have asked this for
quite a while [1][2].

Each EROFS physical cluster contains only one valid standard
Zstandard frame as described in [3] so that decompression can be
performed on a per-pcluster basis independently.

Currently, it just leverages multi-call stream decompression APIs with
internal sliding window buffers.  One-shot or bufferless decompression
could be implemented later for even better performance if needed.

[1] erofs/erofs-utils#6
[2] https://lore.kernel.org/r/Y08h+z6CZdnS1XBm@B-P7TQMD6M-0146.lan
[3] https://www.rfc-editor.org/rfc/rfc8478.txt
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue May 8, 2024
Add Zstandard compression as the 4th supported algorithm since it
becomes more popular now and some end users have asked this for
quite a while [1][2].

Each EROFS physical cluster contains only one valid standard
Zstandard frame as described in [3] so that decompression can be
performed on a per-pcluster basis independently.

Currently, it just leverages multi-call stream decompression APIs with
internal sliding window buffers.  One-shot or bufferless decompression
could be implemented later for even better performance if needed.

[1] erofs/erofs-utils#6
[2] https://lore.kernel.org/r/Y08h+z6CZdnS1XBm@B-P7TQMD6M-0146.lan
[3] https://www.rfc-editor.org/rfc/rfc8478.txt

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240508090346.2992116-1-hsiangkao@linux.alibaba.com
intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue May 8, 2024
Add Zstandard compression as the 4th supported algorithm since it
becomes more popular now and some end users have asked this for
quite a while [1][2].

Each EROFS physical cluster contains only one valid standard
Zstandard frame as described in [3] so that decompression can be
performed on a per-pcluster basis independently.

Currently, it just leverages multi-call stream decompression APIs with
internal sliding window buffers.  One-shot or bufferless decompression
could be implemented later for even better performance if needed.

[1] erofs/erofs-utils#6
[2] https://lore.kernel.org/r/Y08h+z6CZdnS1XBm@B-P7TQMD6M-0146.lan
[3] https://www.rfc-editor.org/rfc/rfc8478.txt

Acked-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
ColinIanKing pushed a commit to ColinIanKing/linux-next that referenced this issue May 9, 2024
Add Zstandard compression as the 4th supported algorithm since it
becomes more popular now and some end users have asked this for
quite a while [1][2].

Each EROFS physical cluster contains only one valid standard
Zstandard frame as described in [3] so that decompression can be
performed on a per-pcluster basis independently.

Currently, it just leverages multi-call stream decompression APIs with
internal sliding window buffers.  One-shot or bufferless decompression
could be implemented later for even better performance if needed.

[1] erofs/erofs-utils#6
[2] https://lore.kernel.org/r/Y08h+z6CZdnS1XBm@B-P7TQMD6M-0146.lan
[3] https://www.rfc-editor.org/rfc/rfc8478.txt

Acked-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240508090346.2992116-1-hsiangkao@linux.alibaba.com
staging-kernelci-org pushed a commit to kernelci/linux that referenced this issue May 10, 2024
Add Zstandard compression as the 4th supported algorithm since it
becomes more popular now and some end users have asked this for
quite a while [1][2].

Each EROFS physical cluster contains only one valid standard
Zstandard frame as described in [3] so that decompression can be
performed on a per-pcluster basis independently.

Currently, it just leverages multi-call stream decompression APIs with
internal sliding window buffers.  One-shot or bufferless decompression
could be implemented later for even better performance if needed.

[1] erofs/erofs-utils#6
[2] https://lore.kernel.org/r/Y08h+z6CZdnS1XBm@B-P7TQMD6M-0146.lan
[3] https://www.rfc-editor.org/rfc/rfc8478.txt

Acked-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240508234453.17896-1-xiang@kernel.org
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants