Download progress freezes, speed decreases until it halts. Unable to open Nautilus and others without closing the app.

[jkrhu]

Describe the bug
I'm downloading the City Sample project through your app. The download progress suddenly freezes at one value after about 5-10 minutes. The system monitor still shows that some progres is being made, but I'm getting around 1-2MB/S instead of 20MB/S when the app in unfrozen. There was a regression on the latest 3.0 version in this regard. The version prior would still have this problem, but much less frequent. I was able to download around 78GB of the 92GB that the project needs, but since 3.0 it's taking way to long to download. Restarting the app fixes it for another 5-10 minutes. But the app has to read the already downloaded 70+ every restart, which is pretty bad. There's also a linked problem that Nautilus doesn't launch at all after the download progress freezes. As well as sometimes restarting the app doesn't show my full library and I can't continue downloading the project again.

To Reproduce
Steps to reproduce the behavior:

1. Select City Sample UE5 Project to download.
2. Wait around 5-10 minutes to see that the download progress doesn't respond.
3. Try opening up Nautilus. The icon switches to a spinning cirle and the app never launches.

Expected behavior
The download progress in app should continue to work and the download speed shouldn't go down, until grinding to a halt. It shouldn't need to be restarted every 15 minutes, reading through dozens of GB every time, to continue the download.

Screenshots
I can't post a screenshot because the screenshot tool doesn't launch as well during the bug happening.

Desktop (please complete the following information):

• OS: Fedora Linux 35, Kernel - 5.16.18-200.fc35.x86_64, GNOME 41.5
• Hardware: CPU: Intel i5-10400, GPU: NVIDIA GeForce RTX 3060, Memory: 16GB

[aknarts]

There is a known issue when the download progress is opened it steals all focus and nothing can be done in a system, but from your description this is not the case.

It is extremely hard to debug something on a large download like this, one thing that I need to do is not hardcode the number of threads used for the download, which will be helpful in some cases.

Other thing that does come to mind is there is a distinct possibility of some files looping in the download queue for ever if they fail to download from any of the CDNs.

I need to take harder look at the asset and engine download more as I am fairly certain both of them can be greatly optimized, especially in the checksuming and resuming downloads.

I have noticed some severe regression in the performance around this particular download, but some of it came from the mere size of the download(the manifest alone is 16MB)...

That being said.. I did have report of people successfully downloading the asset and by the way the asset does not currently work on Linux without some intervention(some cpp assets do not want to compile).

[aknarts]

Would you happen to know if it hogs some resources around the time when the app starts freezing?

[jkrhu]

Thank you for the reply.

Yes, I've noticed the app can steal focus when the download progress is opened, but that's not it.

I've just noticed that nothing new is actually being written to the Vault location. It's stuck at 78GB for the past few attempts and even though the download progresses after restarts, not much is written there. It managed to write only about 77MB during my latest attempt, according to the System Monitor. It might be a problem with this particular project. It's strange that it's happening more frequent after the new version. Maybe some files like the manifest is corrupted after updating to the 3.0.

During the download, it takes consistently around 3GB of RAM and about 2-3% CPU. Doesn't seem to change after the progress freezes.

[aknarts]

Are you downloading to vault or directly creating the project?

There were no significant changes in terms how the download is performed, just functionality added to enable that create project feature. I will dig into it regardless...

Also I envy your 20mb/s speed :D....the CDN is not very fast over here... Another thing it might be doing is extracting some very large asset and get frozen on that for a while, but I guess you are giving it some time.

[jkrhu]

I'm downloading to vault at the default location.

I'll try to delete some files, maybe try to start a new download, then copy stuff over. I'll get back to you if I find something.

Oh yeah, I'm giving it time. It starts at 15-20mb/s, then it goes down to 1-2mb/s when it freezes, then after some time, it grinds to a halt completely. While it has written about 70-100MB. Very strange. And yeah, I'm hardwired with a network switch, so it's pretty speedy, but it's still limited by Epic's servers I think.

[jkrhu]

btw is the extraction bar supposed to be further than the download bar?

[aknarts]

You mean like this?

Yeah.. the download bar is based on size and extraction is based on number of files... Probably should base both on the same thing just so they correspond at least a little.

You can also see more debug output if you run it on the CLI but not sure if it will help... running the download myself now and so far it is running ok, it might get limited on the CDN side at some point but not sure if and when.. The thing is that if there is one large file to be downloaded that will not be greatly multi threaded while at the beginning it downloads all the smaller files in parallel which greatly helps with the speed

[jkrhu]

Yeah, I'm downloading it fresh as well. It goes through it without any problems. I'll try to copy stuff over from a backup and see if that goes smoothly to redownload the rest. I think there might be some files at the end that cause problems like you're saying.

[jkrhu]

Unfortunately, it seems to stop at the exact same spot.

[aknarts]

Well I am running the same and all I ran into is

thread 'Download Pool' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 28, kind: StorageFull, message: "No space left on device" }', src/ui/widgets/download_manager/asset.rs:479:62

No space left on device... so..

1. I should check for needed space before the download
2. I should notify about this issue...

Check df and particularly the tmpfs partition

[aknarts]

The temporary download directory can be changed in preferences to resolve this issue

[jkrhu]

I think that's it! I've changed the temp location and System Monitor shows it's writing GBs now. It's no longer stuck at 70MB. The tmp partition must be limiting it then.

[jkrhu]

It must've been overflowing it to the point where there was no space for other apps. It even crashed Maya once. Very strange.

[aknarts]

Yeah this still needs to get resolved, though it is not an easy thing to do, there is no way of telling in what configuration the chunks will be left on the drive so no way of knowing how big of a disk space it will require... One solution is to drop the temporary download and just do it all in the vault directory, which is what had to be done for the Engine download(for the same reason)

Will think about it, please leave this open.

[aknarts]

Just for fun.. found the offending file...

-rw-r--r-- 1 ue ue 22G Apr  7 02:52 ./DerivedDataCache/Compressed.ddp

facepalming now as that file can be generated and thus should not be in the download at all and would save 22G on the download size...

[jkrhu]

To be fair compiling 22GB worth of shaders is a lot. But still, could've been omitted.

[aknarts]

We are just discussing on Discord if the data is the same for Windows(which this file would be) and on Linux considering dx12 and Vulkan

[jkrhu]

Or right, this file isn't even neeeded for Linux, as the shaders will be compiled for Vulkan.

[aknarts]

Here is a patch to get the CitySample running on Linux https://gist.github.com/aknarts/715097a1579d27f4a1dd363744229f4f
run with

patch -p 0 < CitySampleLinux.diff

Just thought of adding it here :D

[jkrhu]

Actually sent a bug fix to Epic after it failed compiling on VS Code. This is awesome!

[aknarts]

The patch comes from Epic :). Join Unreal Slackers.. highly recommended :D

[jkrhu]

Just did that, thank you!