Axel can't complete a resumed download #233

armanm · 2019-09-10T17:49:40Z

I'm using Axel on OSX 10.14.6 and having trouble resuming an incomplete download. Resuming an interrupted download appears to work until the progress reaches 100% and at that point things seem to just hang – for some unknown reason Axel isn't able to finalise things.

The way I have been experiencing this is when the cable to the external HDD (where I have been downloading to) is accidentally knocked off. Axel appears to be able to pick up from where the download got interrupted but for whatever reason it can't finish the task in the end at which point I have to manually kill Axel.

Here is an example output:

$ axel -a https://.../some-file.xyz
Initializing download: https://.../some-file.xyz
File size: 4891001430 bytes
Opening output file 1201081673.mp4
Starting download

Write error!

Downloaded 3.75457 Gigabyte(s) in 14:30 minute(s). (4520.14 KB/s)

$ axel -a https://.../some-file.xyz
Initializing download: https://.../some-file.xyz
File size: 4891001430 bytes
Opening output file 1201081673.mp4
State file found: 3981082225 bytes downloaded, 909919205 to go.
Starting download

Connection 0 finished
Connection 0 finished
Connection 0 unexpectedly closed
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
Connection 0 finished
[100%] [.................................................#..........................#..........................#.....] [   3.4MB/s] [00:00]^C
Downloaded 867.607 Megabyte(s) in 5:49 minute(s). (2542.91 KB/s)
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

The text was updated successfully, but these errors were encountered:

ismaell · 2019-09-12T10:48:39Z

Version?

A corrupted .st file isn't recoverable. Fixing that would be a major milestone.

arter97 · 2019-11-21T11:07:04Z

You should probably call fsync() to .st files for the filesystem to actually flush the metadata and allow filesystem to try its best to recover it at usable state.

ismaell · 2019-11-21T15:53:48Z

@arter97 That might help, but you can't really recover unless you do the updates incrementally, which we don't. That is, if the disk gets disconnected while the fsync is going on, you're screwed, also you're screwed if the OS decides to write some stuff in between fsync calls.

creynaud · 2020-05-13T11:44:02Z

I had a similar issue, also on macOS, several times with axel version 2.17.7 (darwin18.7.0)
I interrupted, or the download was interrupted because of network issues, 3 big file downloads (15G, 36G, 8G).

After restart, every time, the % got to 100% but got stuck there. Re-launching on my machine (a macbook air macOS 10.14.6 (18G3020)), I got every time a similar output:

$ axel https://download/bigfile.zip
Initializing download: https://download/bigfile.zip
File size: 8954225368 bytes
Opening output file bigfile.zip
State file found: 8954021290 bytes downloaded, 204078 to go.
Starting download

But nothing ever really starts.

Giving both part files to someone else who is not on macOS but on Ubuntu (Axel version 2.17.5 (linux-gnu)), the download finishes fine with axel, for each of my 3 files.

This is the output on Ubuntu:

axel https://download/bigfile.zip
Initializing download: https://download/bigfile.zip
File size: 8954225368 bytes
Opening output file bigfile.zip
State file found: 8954021290 bytes downloaded, 204078 to go.
Starting download
    ,,,,,,,,,, ,......... .......... .......... .......... [  58,9KB/s]
[ 99%] .......... .......... .......... .......... .......... [ 141,7KB/s]
[ 99%] .......... .......... .......... .......... .......... [ 200,3KB/s]
[ 99%] .......... .......... .......... .......... .......... [ 257,5KB/s]
[ 99%] .
Connection 3 finished
,.......
Connection 1 finished
,,,,,,,,..
Connection 2 finished

Downloaded 199,295 Kilobyte(s) in 0 second(s). (268,94 KB/s)

So the part files were not corrupted.

Let me know if I can provide more information, maybe I can run axel again with the parts file to see why it is stuck?

ismaell · 2020-05-13T11:50:07Z

Interesting. Maybe a backtrace of all the threads helps, if this is a simple issue.

creynaud · 2020-05-13T12:28:34Z

I am not using gdb usually for debug, so not 100% this is what you were asking for:

(gdb) info threads

  Id   Target Id                     Frame 
* 1    Thread 0x1703 of process 2380 0x00007fff638a7f2e in ?? () from /usr/lib/system/libsystem_kernel.dylib
(gdb) backtrace
#0  0x00007fff638a7f2e in ?? () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff63833914 in nanosleep () from /usr/lib/system/libsystem_c.dylib
#2  0x00000001077f799c in axel_do ()
#3  0x00000001077fc8e9 in main ()

ismaell · 2020-05-13T13:03:24Z

We need a backtrace of all threads. You can do so with:

thread apply all bt

creynaud · 2020-05-13T15:20:06Z

The "info threads" that is above lists only 1 thread. Doesn't this mean that there is only 1 thread?

Edited: here is the "thread apply all bt" result.

(gdb) thread apply all bt

Thread 2 (Thread 0x2603 of process 5367):
#0  0x00007fff638a7f2e in ?? () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff63833914 in nanosleep () from /usr/lib/system/libsystem_c.dylib
#2  0x000000010000299c in axel_do ()
#3  0x00000001000078e9 in main ()

ismaell · 2020-05-13T18:08:46Z

You're right, my bad. You'll need a version compiled with debugging info or the debugging info file if it was split; that way you'll get more information on the backtrace. A core dump could come handy (you can generate it with the gcore GDB command).

ResetPower · 2021-09-20T14:56:04Z

Hey guys, I've downloaded a big file, but it stuck on 100%, so I used Control+C but I can't resume it...
I don't want to redownload to waste time.
So, I installed Docker on my computer, started a Ubuntu image, and installed ssh on the container,
and downloaded the last part of the big file on Ubuntu,
That's a wonder!

ismaell · 2022-11-07T20:08:55Z

Is this still happening?

s-daveb · 2023-01-01T20:24:37Z

This exact situation is still happening. My router rebooted in the middle of a download, and I resumed it. There were only 280Kb to go, and the program can't complete it.

xanoni · 2023-05-13T00:11:37Z

Still happening ... Ventura 13.3.1

Axel 2.17.11 (darwin21.1.0)

~6GiB download to internal SSD without interruptions, axel -n4 $URL

trevorwang · 2023-06-23T04:14:54Z

(base) ➜  tmp axel -n 5 https://mirrors.tuna.tsinghua.edu.cn/aosp-monthly/aosp-latest.tar
Initializing download: https://mirrors.tuna.tsinghua.edu.cn/aosp-monthly/aosp-latest.tar
File size: 59.666 Gigabyte(s) (64065873920 bytes)
Opening output file aosp-latest.tar
State file found: 64065693994 bytes downloaded, 179926 to go.
Starting download

�stuck on last bytes

ismaell · 2023-06-25T00:03:10Z

"me too"-ing doesn't help; this seems to be Apple-specific, so users need to step up and do the debugging, otherwise it won't be fixed, I don't have magic powers.

s-daveb · 2023-06-26T15:28:19Z

I compiled the source with debug symbols, attached the llvm/clang debugger, ran a resumed download, and after it printed "Connection 4 finished", I got the following trace.

(lldb) thread backtrace all
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007ff805bd2fbe libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007ff805ac5585 libsystem_c.dylib`nanosleep + 196
    frame #2: 0x000000010000400b axel`axel_sleep(delay=(tv_sec = 0, tv_nsec = 100000000)) at sleep.c:50:16
    frame #3: 0x0000000100003a25 axel`axel_do(axel=0x000000010100a400) at axel.c:523:7
    frame #4: 0x0000000100009520 axel`main(argc=<unavailable>, argv=<unavailable>) at text.c:420:3
    frame #5: 0x00007ff8058b541f dyld`start + 1903

I'm speculating, but it looks "waiting for connection" part of the loop (axel.c:523) continues running after the "Connection is finished" message is printed.

I tried setting a breakpoint past line 523, at 530, and another one at line 916 but it looks like it's never hit.

s-daveb · 2023-06-26T15:46:39Z

I want to add that on the latest macOS SDK, the application wouldn't even compile because I think the default version of C is C11. I had to make several small changes to printf-style format specifiers, and add _Atomic specifiers. See PR #408

Maybe compiling it with C11 support on Linux will also lead to the same behavior? Sorry to speculate, but I'm wondering if this could be something due to the default behavior of the macOS SDK / C runtime, and not the system calls, or the OS itself.

ismaell · 2023-06-26T21:42:56Z

How can we check the version of the SDK? the proper way would be to add -std=c99 to the compiler flags, but I'm not sure it's safe to add it unconditionally...

MarisaDAZA · 2023-07-28T05:52:32Z

I also meet this problem, on axel 2.17.11. Before it stuck, new connection always started in the middle of two old connections. But the old ones can't continue. I think that's the reason.
However axel on other platform can finish this download.

ismaell self-assigned this Sep 12, 2019

ismaell added the feature request label Sep 12, 2019

ismaell added this to the v2.18 milestone Sep 12, 2019

ismaell modified the milestones: v2.18, v2.19 May 13, 2020

ismaell added bug Non-Free Platform The issue affects a non-free OS, which makes it difficult to provide support and removed feature request labels May 13, 2020

Ionic mentioned this issue Dec 30, 2020

Sleep time overflow with > 5 connections #338

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Axel can't complete a resumed download #233

Axel can't complete a resumed download #233

armanm commented Sep 10, 2019

ismaell commented Sep 12, 2019

arter97 commented Nov 21, 2019

ismaell commented Nov 21, 2019

creynaud commented May 13, 2020

ismaell commented May 13, 2020

creynaud commented May 13, 2020 •

edited

ismaell commented May 13, 2020

creynaud commented May 13, 2020 •

edited

ismaell commented May 13, 2020

ResetPower commented Sep 20, 2021

ismaell commented Nov 7, 2022

s-daveb commented Jan 1, 2023

xanoni commented May 13, 2023

trevorwang commented Jun 23, 2023

ismaell commented Jun 25, 2023

s-daveb commented Jun 26, 2023 •

edited

s-daveb commented Jun 26, 2023 •

edited

ismaell commented Jun 26, 2023 •

edited

MarisaDAZA commented Jul 28, 2023

Axel can't complete a resumed download #233

Axel can't complete a resumed download #233

Comments

armanm commented Sep 10, 2019

ismaell commented Sep 12, 2019

arter97 commented Nov 21, 2019

ismaell commented Nov 21, 2019

creynaud commented May 13, 2020

ismaell commented May 13, 2020

creynaud commented May 13, 2020 • edited

ismaell commented May 13, 2020

creynaud commented May 13, 2020 • edited

ismaell commented May 13, 2020

ResetPower commented Sep 20, 2021

ismaell commented Nov 7, 2022

s-daveb commented Jan 1, 2023

xanoni commented May 13, 2023

trevorwang commented Jun 23, 2023

ismaell commented Jun 25, 2023

s-daveb commented Jun 26, 2023 • edited

s-daveb commented Jun 26, 2023 • edited

ismaell commented Jun 26, 2023 • edited

MarisaDAZA commented Jul 28, 2023

creynaud commented May 13, 2020 •

edited

creynaud commented May 13, 2020 •

edited

s-daveb commented Jun 26, 2023 •

edited

s-daveb commented Jun 26, 2023 •

edited

ismaell commented Jun 26, 2023 •

edited