Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exception thrown in worker thread: class dwarfs::runtime_error: lzma_stream_encoder #224

Closed
mindfocus opened this issue May 30, 2024 · 2 comments
Assignees
Labels
bug Something isn't working fixready
Milestone

Comments

@mindfocus
Copy link
Contributor

mindfocus commented May 30, 2024

use bat in windows 11

@echo off
@chcp 65001
echo 开始处理...
@setlocal
@set "inputFloder=%~1"
@set "outputFile=%~1.dwarfs"
E:\dwarfs-0.9.9-Windows-AMD64-ricepp\bin\mkdwarfs.exe -l 9 -i "%inputFloder%" -o "%outputFile%" --log-level trace
@endlocal
echo 处理结束.
@pause

T 01:29:12.376901 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 462,336 from chunkable offset 4,989,791
T 01:29:12.377074 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 462,592 from chunkable offset 4,990,047
T 01:29:12.377270 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 462,848 from chunkable offset 4,990,303
T 01:29:12.377443 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 463,104 from chunkable offset 4,990,559
T 01:29:12.377616 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 463,360 from chunkable offset 4,990,815
T 01:29:12.377790 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 463,616 from chunkable offset 4,991,071
T 01:29:12.377965 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 463,872 from chunkable offset 4,991,327
T 01:29:12.378167 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 464,128 from chunkable offset 4,991,583
T 01:29:12.378516 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 464,384 from chunkable offset 4,991,839
T 01:29:12.378632 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 464,640 from chunkable offset 4,992,095
T 01:29:12.378712 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 464,896 from chunkable offset 4,992,351
T 01:29:12.378823 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 465,152 from chunkable offset 4,992,607
T 01:29:12.379012 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 465,408 from chunkable offset 4,992,863
T 01:29:12.379109 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 465,664 from chunkable offset 4,993,119
T 01:29:12.379198 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 465,920 from chunkable offset 4,993,375
T 01:29:12.379303 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 466,176 from chunkable offset 4,993,631
T 01:29:12.379379 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 466,432 from chunkable offset 4,993,887
T 01:29:12.379461 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 466,688 from chunkable offset 4,994,143
T 01:29:12.379548 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 466,944 from chunkable offset 4,994,399
T 01:29:12.379666 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 467,200 from chunkable offset 4,994,655
T 01:29:12.379837 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 467,456 from chunkable offset 4,994,911
T 01:29:12.380011 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 467,712 from chunkable offset 4,995,167
T 01:29:12.380185 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 467,968 from chunkable offset 4,995,423
T 01:29:12.380374 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 468,224 from chunkable offset 4,995,679
T 01:29:12.380500 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 468,480 from chunkable offset 4,995,935
T 01:29:12.380593 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 468,736 from chunkable offset 4,996,191
T 01:29:12.380678 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 468,992 from chunkable offset 4,996,447
T 01:29:12.380761 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 469,248 from chunkable offset 4,996,703
T 01:29:12.380842 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 469,504 from chunkable offset 4,996,959
T 01:29:12.380961 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 469,760 from chunkable offset 4,997,215
T 01:29:12.381135 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 470,016 from chunkable offset 4,997,471
T 01:29:12.381314 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 470,272 from chunkable offset 4,997,727
F 01:29:12.381179 [worker_group.cpp:284] exception thrown in worker thread: class dwarfs::runtime_error: lzma_stream_encoder [lzma.cpp:152]
T 01:29:12.381453 [segmenter.cpp:1057] appending 256 bytes to block 1 @ 470,528 from chunkable offset 4,997,983
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
writing: \?\E:\BaiduYunDownload\06.新东方国学\诗经\诗经.12.黍离.mp3
5 dirs, 0/0 soft/hard links, 186/186 files, 0 other
original size: 1.242 GiB, hashed: 367.2 MiB (46 files, 1.191 MiB/s)
scanned: 1.062 GiB (163 files, 781 KiB/s), categorizing: 0 B/s
saved by deduplication: 183.6 MiB (23 files), saved by segmenting: 90.52 KiB
filesystem: 64.21 MiB in 1 blocks (8 chunks, 6/163 fragments, 163 inodes)
compressed filesystem: 0 blocks/0 B written
███▍ ▏ 3% 🌕
\?\E:\BaiduYunDownload\06.新东方国学\诗经\诗经.12.黍离.mp3 █▉ ▏1.332 MiB/s
[compressing] 0 B/s

@mhx
Copy link
Owner

mhx commented May 30, 2024

Hi and thanks for the report!

You've found a bug that's been in the code for a really long time. The tl;dr is that LZMA encoder objects can be cloned and clones would internally still point to data from the original instance. Depending on whether or not the original instance is freed, if a new instance gets allocated at the same address, or how quickly the memory allocator reuses the freed memory, this would cause lzma_stream_encoder to be called with invalid options.

Fortunately, it's an easy fix. I'll prepare a new release ASAP.

mhx added a commit that referenced this issue May 30, 2024
When cloning an `lzma_block_compressor`, the cloned `filters_` would
point to the original `opt_lzma_` instead of the cloned one. While
this is obviously a bug, the code would usually work and only
occasionally bail out with an lzma error.

This change does away with having `filters_` as a member and rather
builds the array each time `compress` is called. The overhead is
completely negligible.
mhx added a commit that referenced this issue May 30, 2024
When cloning an `lzma_block_compressor`, the cloned `filters_` would
point to the original `opt_lzma_` instead of the cloned one. While
this is obviously a bug, the code would usually work and only
occasionally bail out with an lzma error.

This change does away with having `filters_` as a member and rather
builds the array each time `compress` is called. The overhead is
completely negligible.
@mhx mhx self-assigned this May 30, 2024
@mhx mhx added the bug Something isn't working label May 30, 2024
@mhx mhx added this to the v0.9.10 milestone May 30, 2024
@mhx mhx added the fixready label May 30, 2024
@mhx
Copy link
Owner

mhx commented May 30, 2024

Fixed in v0.9.10.

@mhx mhx closed this as completed May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fixready
Projects
None yet
Development

No branches or pull requests

2 participants