Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS client hangs when downloading corrupt compressed blocklists #4702

Closed
GaryElshaw opened this issue Feb 1, 2023 · 20 comments · Fixed by #4705
Closed

macOS client hangs when downloading corrupt compressed blocklists #4702

GaryElshaw opened this issue Feb 1, 2023 · 20 comments · Fixed by #4705

Comments

@GaryElshaw
Copy link
Contributor

GaryElshaw commented Feb 1, 2023

What is the issue?

This bug appears to be back, automated blocklist updating is seemingly not happening, again.

SCR-20230201-x9v

Edit: An added bonus. When manually updating, 'Processing blocklist...' continues into infinity and the cancel button is greyed out.

SCR-20230202-4y

Which application of Transmission?

macOS app

Which version of Transmission?

4.0.0-beta.3.dev (724a0f4)

@GaryElshaw GaryElshaw changed the title Automatic blocklist updating Automatic blocklist updating isn't working Feb 1, 2023
@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

Duplicate of #2272, I think. It's had a pr welcome label since June 2022 😸

@ckerr ckerr closed this as completed Feb 1, 2023
@GaryElshaw
Copy link
Contributor Author

Sadly, no :-( It had been resolved some time ago, but is back. This is a potential blocker. Sucks, i know.

@ckerr ckerr reopened this Feb 1, 2023
@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

What is the URL of the blocklist that you're using?

@GaryElshaw
Copy link
Contributor Author

What feels like the project default: https://github.com/Naunter/BT_BlockLists

@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

That is not the URL in your screenshot.

I'm trying to reproduce the bug; could you please paste the URL that you're using when you encounter this failure 🙏

@ckerr ckerr added needs clarification More info is needed before work can proceed scope:mac labels Feb 1, 2023
@GaryElshaw
Copy link
Contributor Author

@ckerr ckerr added confirmed bug:regression and removed needs clarification More info is needed before work can proceed labels Feb 1, 2023
@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

Confirmed.

Seems to require .gz in order to trigger the bug.

https://github.com/Naunter/BT_BlockLists/raw/master/bt_blocklists works

https://github.com/Naunter/BT_BlockLists/raw/master/bt_blocklists.gz hangs forever

@ckerr ckerr changed the title Automatic blocklist updating isn't working macOS client can't download gzipped blocklists Feb 1, 2023
@nevack
Copy link
Member

nevack commented Feb 1, 2023

Naunter/BT_BlockLists@master/bt_blocklists.gz (raw)

Ok, I have tried to debug it myself.
This file is invalid tar archive.
@ckerr can you confirm?
Checksum:
0d25e429de8f1ac5cc51ab3bb99aee326297b545a019e0a58fffc8ade7ae67db bt_blocklists.gz
File size:
680318 bytes

@nevack
Copy link
Member

nevack commented Feb 1, 2023

I agree we need to improve this process and handle errors more graceful.

@nevack
Copy link
Member

nevack commented Feb 1, 2023

@ckerr You can close this one.
I've filled Naunter/BT_BlockLists#16

@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

@nevack looks like it's hanging when callling /usr/bin/tar --list --file $foo and then waiting on [tarList waitUntilExit]. Should we handle that better when fed an invalid archive?

@nevack
Copy link
Member

nevack commented Feb 1, 2023

@nevack looks like it's hanging when callling /usr/bin/tar --list --file $foo and then waiting on [tarList waitUntilExit]. Should we handle that better when fed an invalid archive?

Yep, I see the same hang on waitUntilExit.
I'll take a look what we can do.

I think we should use some lightweight 3rd-party library for working with archives, current implementation (which is my refactor) is a burden :(

@nevack
Copy link
Member

nevack commented Feb 1, 2023

As per https://stackoverflow.com/questions/2001709/how-to-check-if-a-unix-tar-gz-file-is-a-valid-file-without-uncompressing
we cannot just test archive before listing.
Also, it's really fast check when done by hand in terminal.
I think piping it with NSPipe kills all perfomance.

@nevack
Copy link
Member

nevack commented Feb 1, 2023

tar --list --file bt_blocklists.gz | wc -lc gives me 99457 lines and 4030980 chars.

@GaryElshaw
Copy link
Contributor Author

There are three things. There is the automation/update of the file; how that file is manually updated; but also that the dialog window cannot cancel the process of manually updating.

@nevack
Copy link
Member

nevack commented Feb 1, 2023

As per stackoverflow.com/questions/2001709/how-to-check-if-a-unix-tar-gz-file-is-a-valid-file-without-uncompressing we cannot just test archive before listing. Also, it's really fast check when done by hand in terminal. I think piping it with NSPipe kills all perfomance.

Yup, piping to null fixes this hang.
Let's add check for archive validity.

@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

Yup, piping to null fixes this hang.

That's what I tried first via the NSTask output property, but wasn't sure how to get the first filename when doing that...

I think we should use some lightweight 3rd-party library for working with archives, current implementation (which is my refactor) is a burden :(

Agreed! I would love if this was done in core so that it works in non-macOS clients too.

Something like KArchive but without the dependency chain...

Maybe https://github.com/libarchive/libarchive ?

nevack added a commit to nevack/transmission that referenced this issue Feb 1, 2023
Recently we discovered we do not properly handle broken tar archives.

The problem is running tar (in linting mode) on invalid archive and
piping stdout to NSPipe introduces a hang as NSPipe cannot process
such huge data amounts.

Fixing this problem by adding a precheck with tar (listing mode) with
piping stdout to /dev/null.

Fixes: transmission#4702
@nevack
Copy link
Member

nevack commented Feb 1, 2023

That's what I tried first via the NSTask output property, but wasn't sure how to get the first filename when doing that...

I decided to run listing twice - with stdout > dev/null and then with stdout > NSPipe.
Tested locally with mentioned link to gz.

@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

Yep, WfM too. Thanks for the fast fix @nevack ! 🚀

nevack added a commit to nevack/transmission that referenced this issue Feb 1, 2023
Recently we discovered we do not properly handle broken tar archives.

The problem is running tar (in linting mode) on invalid archive and
piping stdout to NSPipe introduces a hang as NSPipe cannot process
such huge data amounts.

Fixing this problem by adding a precheck with tar (listing mode) with
piping stdout to /dev/null.

Fixes: transmission#4702
@ckerr ckerr changed the title macOS client can't download gzipped blocklists macOS client hangs when downloading corrupt compressed blocklists Feb 1, 2023
@ckerr ckerr added this to the 4.0.0 milestone Feb 1, 2023
@ckerr
Copy link
Member

ckerr commented Feb 1, 2023

@GaryElshaw you're right; this is a blocker. Thanks for reporting this. Also, thanks for persisting when I initially misunderstood the report 👍

@ckerr ckerr reopened this Feb 1, 2023
ckerr pushed a commit that referenced this issue Feb 1, 2023
Recently we discovered we do not properly handle broken tar archives.

The problem is running tar (in linting mode) on invalid archive and
piping stdout to NSPipe introduces a hang as NSPipe cannot process
such huge data amounts.

Fixing this problem by adding a precheck with tar (listing mode) with
piping stdout to /dev/null.

Fixes: #4702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

3 participants