Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

[Linux] Client starts a way too long #661

Open
maximvl opened this Issue · 81 comments
@maximvl

On my system native client launches nearly 50 seconds while under wine its around 15 seconds.
I believe it should be less or equal in worst case.
btw after fixed hero selection crash this bug is most annoying.

Processor Information:
    Vendor:  GenuineIntel
    Speed: 2501 Mhz
    4 logical processors
    2 physical processors
    HyperThreading:  Supported
    FCMOV:  Supported
    SSE2:  Supported
    SSE3:  Supported
    SSSE3:  Supported
    SSE4a:  Unsupported
    SSE41:  Supported
    SSE42:  Supported

Network Information:
    Network Speed:  

Operating System Version:
    Ubuntu 13.04 (64 bit)
    Kernel Name:  Linux
    Kernel Version:  3.8.0-27-generic
    X Server Vendor:  The X.Org Foundation
    X Server Release:  11304000
    X Window Manager:  Xfwm4
    Steam Runtime Version:  steam-runtime-release-i386_2013-06-28

Video Card:
    Driver:  Intel Open Source Technology Center Mesa DRI Intel(R) Ivybridge Mobile x86/MMX/SSE2

    Driver Version:  3.0 Mesa 9.2.0-devel
    Desktop Color Depth: 24 bits per pixel
    Monitor Refresh Rate: 60 Hz
    VendorID:  0x8086
    DeviceID:  0x166
    Number of Monitors:  1
    Number of Logical Video Cards:  1
    Primary Display Resolution:  1366 x 768
    Desktop Resolution: 1366 x 768
    Primary Display Size: 13.54" x 7.64"  (15.51" diag)
                                            34.4cm x 19.4cm  (39.4cm diag)
    Primary VRAM Not Detected

Sound card:
    Audio device: Intel PantherPoint HDMI

Memory:
    RAM:  3655 Mb

Miscellaneous:
    UI Language:  English
    LANG:  en_US.UTF-8
    Microphone:  Not set
    Total Hard Disk Space Available:  465592 Mb
    Largest Free Hard Disk Block:  382425 Mb

Installed software:

Recent Failure Reports:

@maximvl maximvl referenced this issue
Closed

Long map loading #652

@davidw-valve
Owner

I take it you are talking about initial startup time to the lobby rather than map loading time?

What kind of storage device do you have your Steam library on? Do you have it configured with Ubuntu's encryption?

@f4hy

davidw, do you have any info on which file system performs best for dota/steam library? My startup time for both initial startup time and map loading time seem slower than on win. I wonder if it is a filesystem choice issue?

@maximvl

I measured time from dota logo and "loading" text appearing to main menu and music. Steam is on regular notebook hdd, without any encryption. dota launch options: -novid -console -international

@maximvl

f4hy, dota starts faster on wine, so filesystem isnt an issue.

@Terin

In a test I did just now, it took 29.5 seconds to load Dota 2 on Windows 8, installed on a 7200 RPM HDD. On Arch Linux, it took 1 minute and 15 seconds to load on a 120GB SSD. I experienced similar launch times on Ubuntu 12.04, although I don't have it installed at the moment to give exact times.

There is definitely a significant bug affecting loading performance on the Linux client.

@mastertheknife

Same here, loading the map for the first time takes really long time.

@beamhardbc

The loading time is so bad. I rather play in 11 fps rather than restart DotA to make it 40 again in view of abandon.

@maximvl

Map loads longer for the first time on windows too, but this ticket is about loading to the main menu.

@yoyosan

I have the same problem since the first release. I'm using a 120GB SSD for the Steam Library(for both Linux and Windows). The loading time in Linux is ~1min and a half. In Window 7, it's ~10 seconds.

Any idea why?

Specs https://gist.github.com/yoyosan/6049783

@davidw-valve
Owner

Such long startup times are not expected behavior. What filesystem are you using? How much CPU time does the dota_linux process consume during this loading? How much filesystem activity is there?

@beamhardbc

OK, I am a Linux noob.

1.I just made a backup of DotA.
2. Deleted original game.
3.Installed the game in my unmounted partition with SteamLibrary

Now the game loading is halved. I am not quite sure but I think there was a 50 mb update in between this process. Anyways do give this a try. both my partitions are ext4 filesystem i dont have a single drop of Windows on my hard disk, it is linux through and through after dota 2 release. :D

so i did 3 restarts of client, i got 50 secs loading on avg. compared to 130 secs when game was installed in .steam root folder.
the map however loads inconsistently load time ranging from 20 secs to 120 secs. i AM talking about first load.

so @davidw-valve 's query actually helped.
https://gist.github.com/beamhardbc/6060799

@gensss

same thing, wine ver starts faster

cpu isnt bottlenecked as far as i can see (single thread heavier then others at start, later no core 100%)
disk is relatively quiet

http://pastebin.com/LmWSteGu
perf statistics for first 30sec of loading

fx-8320
nvidia 8800 gt
ext4
kernel 3.10.5

i can give more perf stats or trace calls if you want

@jasiolpn

I think this problem is connected with issue #652. I am using ext4 file system. Dota needs about 1 minute to load.

@yoyosan

@davidw-valve
Sorry for being a bit ignorant about the amount of details reported. Here is more about my system.

I have a separate partition for the Steam library, on a 120 GB SSD.
The file system used for the Steam library is NTFS, mounted like this:

/dev/sda6 /media/MEDIA7 ntfs-3g defaults,user,locale=en_US.utf8,exec,uid=1000,gid=1000 0 0

This partition is also used on Windows so that I have a common storing location for both OS-es.

Since during Dota startup I can't Alt-Tab until it finishes loading the game, I've made a little bash script that logs the process and disk IO information to a text file(during loading of course). Here is the output: https://gist.github.com/yoyosan/6199514

Also, I've did more tests regarding loading time and it is ~1min 17 seconds.

If there is something else you need, please ask.

LE:
I've generated better stats by including top command output here.

@fasmat

@yoyosan NTFS performace is generally very bad in linux. Have you tried using ext4, btrfs or another more usual linux file system?

@Terin

@fasmat I have an ext4 partition and the same loading time, so it's definitely not exclusive to NTFS partitions.

@deri82

start the game with command -novid

That saves some time.

@fasmat

@deri82
I do, but still it takes at least 3 times as long to start the game in Linux than in Windows, while neither my CPU nor my file system are used more than 50%.

@fasmat

Did not improve with today's update. Starting Dota 2 still takes 30+ seconds from double clicking the icon to main menu with -novid as launch parameter.

@davidw-valve

Does running with -nosound and/or -nojoy make things any faster?

@fasmat

Unfortunately not by a great amount:

-novid:

  • 8 seconds to full screen black
  • another 20 seconds to main menu (first start after boot takes ~ 3 seconds longer)

-novid -console:

  • 8 seconds to full screen black
  • another 19 seconds to main menu

-novid -console -nojoy -nosound:

  • 8 seconds to full screen black
  • another 18 seconds to main menu
@maximvl

davidw-valve, I suddenly found that test client starts much faster, like in windows, test this anybody please, hope this could help to solve the problem.

@Weegee

For me it's like this:

DOTA 2 main client

  • 38 seconds from pressing the "Play" button in Steam to the main menu
  • 40 seconds until I'm in a local bot match after pressing "Start Game"

DOTA 2 test client

  • 32 seconds from pressing the "Play" button in Steam to the main menu
  • 40 seconds until I'm in a local bot match after pressing "Start Game"

Here's my system information, I use -console -novid -high -windowed -noborder -dev -threads 4 as launch options for both the main and the test client.

All in all it doesn't really need that much longer than on Windows to start the game, but I'll test it there as well just to be sure.

@maximvl

Kernel Version: 3.10.7-1-ARCH

Hmm, second time I see good launching time with kernel > 3.8, probably this is the reason.

@mastertheknife

Might be kernel thing then. With stock fedora 19 kernel (3.9.9):
9.5 seconds from Play button to intro vid
5 seconds after intro video until main menu.

@maximvl

I have updated kernel to latest 3.10.9, launch time still around one minute for main client.

@Weegee

Okay, I have to take back what I said earlier. Here are my Windows results:

DOTA 2 main client

  • 9 seconds from pressing the "Play" button in Steam to the main menu
  • 28 seconds until I'm in a local bot match after pressing "Start Game"

Also, I forgot to mention that my DOTA 2 files under Linux are on a NTFS drive.

@deri82

I haven't calculated timings, but dota does take load quite long time (steam too). But system resources show that dota uses over gigabyte of memory.

I have hdd in use, basically dota loading speed is 1 gigabyte/50megabytes a second transferring speed of hdd. Whatever it equals :D

@maximvl

But system resources show that dota uses over gigabyte of memory.

If you are talking about ram its not equal the actual size of data loaded from files, its better to use time measurements.

btw, davidw-valve is it possible to add some kind of debug key to dota launch options to turn on internal timers and get some loading time measures/stats?

@deri82

I guess you are right. But that's a lot of data to be loaded in ram. That takes quite long to load on regular hdd.

@fal3ur3

+1 on this issue; for me however, the initial loading to the lobby can take up to 60 seconds under Linux. With the same machine, under Windows, loading time varies but is usually under 20 seconds. Cross referencing #488 because I definitely believe they are related; in both cases, Dota consumes no more than 25% CPU usage at any time, and disk usage is as follows:
stops after about 20-30 seconds while loading lobby, then RIGHT before lobby shows up, disk usage jumps up again for a few seconds. This is similar behavior to the issue I linked above, but with different load times, etc.
With each update that is released, the Linux issues seem to get more severe, more frequent, or simply more are created.

@deri82

If I look at the console information what I get I have no idea what it means. Same with steam.

How about changing steam and dota console information that way that we can understand what it actually says?

For example my dota informs me that i have missing some textures or something. I don't know if it's serious issue or not. Don't remember the exact error.

@deri82

I just checked my dota2 console

I have these errors there, just picked some random lines.

Warning: Setting unsupported dynamic pixelshader combo WRITEWATERFOGTODESTALPHA!
Receiving uncompressed update from server
Redownloading all lightmaps
C:Gamerules: entering state 'DOTA_GAMERULES_STATE_WAIT_FOR_PLAYERS_TO_LOAD'
Requesting texture value from var "$basetexture" which is not a texture value (material: dev/halo_add_to_screen)
C:Gamerules: entering state 'DOTA_GAMERULES_STATE_HERO_SELECTION'
Randoming npc_dota_hero_rattletrap!
Particles: Missing 'range_display'
Model heroes/omniknight/omniknight.mdl over budget: 3981 tris, 7 batches
Model heroes/rattletrap/rattletrap.mdl over budget: 6580 tris, 6 batches

And the font is somehow small. I have to look very closely to see what's written on 1920x1080.

@deri82

I just started my steam client. It said to me that it took 17 seconds to load (I have to type password - is that calculated too?).

I think that's pretty slow. Starting steam should not take that long. It's not that complicate application.

@dietrmat

This issue is very annoying. Often it takes so long when the game locks up and I have to reboot, that I get an abandoned game because starting the game takes so long.

I stopped the time:
1. From hitting play until video valve video it takes: 17 sec
2. From skipping the video until game menu it takes another: 112 sec

@deri82

Starting steam:"System startup time: 137,31 seconds"

???

You can skip the video by setting starting launch options:"-novid", without " marks. If you don't have joypad add also -nojoy command.

@dietrmat

Retestet with latest stable Kernel 3.11 (using ext4 encrypted with LUKS), no improvements. From play button to menu: 142 sec.

@vrodic

I think that adding +mat_autoload_glshaders 0 to startup will help the startup time (as discussed in #774).

@maximvl

@vrodic yes, this option reduced loading time twice, and I had no crashes in 4 hours, thx!

@dietrmat

@vrodic Although setting the option reduced Dota's initial loading time for reaching the menu, the initial loading for reaching a game is drastically increased, with the result that the game time out before I could enter it.

@vrodic

Your PC might be too slow, and you must use the regular value for that option then. I don't experience such drastic issues. But I mostly use the Windows client now, since Linux version is just not that good in general (higher memory use, longer startup time, lower FPS, on my HD4000 system).

@maximvl

@dietrmat I dont have timeout issue with that option. (Intel HD4000 video)
@vrodic hm, I experience more fps, and same loading time with autoload_glshaders 0.

@yoyosan

I've re-formatted the partition to btrfs(same goes for ext4) and I get Windows load times.

@Weegee

Okay, I made some new tests with a spare HDD I found, here's a benchmark for it from GNOME Disks.

I created three partitions on the HDD (NTFS, ext4, BTRFS) and installed DOTA 2 Test on all three, then tested several times how long it takes from pressing play in Steam to the DOTA 2 main menu and from pressing Start Game in the Local Bot Game tab until I could freely move my character. The results I got:

NTFS:

  • Play game -> Main menu: 24s
  • Start bot game -> In-game: 36s

ext4:

  • Play game -> Main menu: 21s
  • Start bot game -> In-game: 21s

btrfs:

  • Play game -> Main menu: 19s
  • Start bot game -> In-game: 22s

My launch options are schedtool -I -e %command% -console -novid -high -dev -threads 4 +mat_autoload_glshaders 0, also my current System Information and config.cfg from DOTA 2 Test.

I think I might switch my main data partition from NTFS to ext4 now ...

@maximvl

@Weegee thanks for information, looks like problem was in shaders, probably it should be added to linux version options by default or so, feel free to close issue then.

@felipec

For me it takes two full minutes, actually more like 2:10. With +mat_autoload_glshaders it starts in around 45s, so it's definitely better. -novid shaves an additional 15 seconds.

This is on an ext4 partition and an SSD able to read at 410 MB/sec.

The default is really annoying, fortunately there's the mat_autoload_glshaders option.

@nh2
nh2 commented

@maximvl It is still weird that Wine vs. native can make such a big difference. That really sounds like a bug or unoptimized code path somewhere.

Could you make some more comprehensive benchmark, comparing Wine vs. native startup times across different file systems, and with and without the options discussed in this issue?

Also, it might make sense to run eg. dstat while starting Dota - that gives a nice output every second that you could post here, e.g.:

dstat --cpu --disk --mem --net --proc --io --aio --fs --int --sys
----total-cpu-usage---- -dsk/total- ------memory-usage----- -net/total- ---procs--- --io/total- async --filesystem- ----interrupts--- ---system--
usr sys idl wai hiq siq| read  writ| used  buff  cach  free| recv  send|run blk new| read  writ| #aio|files  inodes|  45    46    47 | int   csw 
 21   7  72   0   0   0|  45k   61k|4194M  431M 2065M 1179M|   0     0 |  0   0 0.5|2.33  1.52 |   0 | 8864    391k|   0     0    35 | 744  1436 
  4   5  90   0   0   0|   0     0 |4194M  431M 2065M 1179M|   0     0 |  0   0   0|   0     0 |   0 | 8896    391k|   0     0    22 |3316  6436 
  3   4  93   0   0   0|   0    56k|4194M  431M 2065M 1179M|   0     0 |  0   0   0|   0  10.0 |   0 | 8896    391k|   1     0    25 |3288  6391 
  4   5  91   0   0   0|   0   412k|4194M  431M 2065M 1179M|   0     0 |1.0   0   0|   0  49.0 |   0 | 8896    391k|   0     0    27 |3266  6257
@nh2
nh2 commented

Some relevant blog post with other related benchmarks: http://vrodic.blogspot.co.uk/2013/08/dota-2-performance-linuxnative-vs.html

@deri82

I have experienced loading speed of over 1 minute of dota on native client to the play screen.

There is a speed test on steam. I dont't know how often this is taken. Or actually when.

But I set the variable on the speed my cpu currently is on dota starting settings:

"RDTSC_FREQUENCY=3500.00"

I had one dota 2 open, I closed it and started steam and dota again and the game loaded fully in about 20seconds. No changes on pc specs.

That's great improvement even I had started dota a bit earlier once. That havent been effecting starting speed much before. So it cannot be the reason.Setting the envirioement is the reason.

@maximvl

@nh2 have you tried "+mat_autoload_glshaders 0" startup option?

@deri82

Yes, but the loading speed has been bad even with that option. I am currently using that too but I have some lag there and here.

@deri82

Try yourself adding cpu speed to the variable on dota launching setting.

@tghosgor

I also suffer from this issue, dota start-up is slow, game load is also slow compared to windows, first load of inventory pages just hangs the game for a few seconds also. I will write if that cpu speed flag fixes it.

@Aviramk

I can confirm deri82's fix.. but what about a real fix?

@chrylis

The manual CPU setting worked but was broken by an update in December.

@deri82

I don't have timing measures. But I have recently tried wine version of dota and native version.

Wine consumes around 600 000kb memory when it's loading screen. Native uses 1.3 gigabytes of memory at same place. And even more at game. This calculations don't include shared memory or steam itself.

The loading speed of wine version feels a lot faster.

@nh2
nh2 commented
@vhda

The launch option "+mat_autoload_glshaders 0" greatly increases startup time. In my case it goes from 1'40" down to 30". To start a bot game it takes an extra 45", but I think this is normal.

I also noticed that the "-threads 2" and "-high" launch options do not work on Linux. When using "top" the dota process only uses one CPU and priority is still "0", which is the default. Could anyone confirm if the windows version running under wine uses more than one CPU?

One other thing: from my analysis after removing "+mat_autoload_glshaders 0" the game no longer takes the original 1'40"! Does this option become permanent?

@deri82

I just wanted to try that command

"+mat_autoload_glshaders 0"

AND IT DOES decrease the loading speed of dota a lot.

Normally dota consumes 1.3 gigabytes of memory when it's fully loaded.
Now using this parameter it's only 834 000kb. The difference is around 500 megabytes!!

@deri82

BUT

"Wine consumes around 600 000kb memory when it's loading screen. "

That's around 200 megabytes more than what wine uses.

@nh2

That's around 200 megabytes more than what wine uses.

Are you actually sure that the Wine and the native version do the same? Maybe the Wine version loads less things for some reason? Also you haven't answered my 32-bit question yet.

@deri82

I ALSO EXPERIENCE HUGE FPS BOOST!

@deri82

I have native/vanilla crossover install. I don't know what wine it uses.

@vhda

Assuming that you have a Debian based GNU/Linux operating system, you should do the following:

dpkg -l crossover\*

For example, I have wine installed and this is what I get:

$ dpkg -l wine\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                    Version                  Architecture             Description
+++-=======================================-========================-========================-====================================================================================
ii  wine                                    1.6.2-7                  amd64                    Windows API implementation - standard suite
in  wine-bin                                <none>                   amd64                    (no description available)
un  wine-doc                                <none>                   <none>                   (no description available)
ii  wine32                                  1.6.2-7                  i386                     Windows API implementation - 32-bit binary loader
ii  wine64                                  1.6.2-7                  amd64                    Windows API implementation - 64-bit binary loader
un  wine64-bin                              <none>                   <none>                   (no description available)

I have both amd64 (64 bits) and i386 (32 bits) packages installed.

@deri82

It's i386.

@nh2

It's i386.

That's what might explain the difference in used RAM (since your native version is probably 64 bit), but that's more of a guess than for sure; it could also be that the Wine version loads less (e.g. textures or so), or that it is a bug.

@davidw-valve Any idea what's going on?

@chrisforbes

@nh2 native version is 32bit too.

@deri82

I have noticed something very interesting.

Setting +mat_autoload_glshaders 0 launch option decreased the memory usage alot.
I had steam segfault error, i tried to reset steam (steam -reset), didnt help. Then I removed appcache directory.

And now I run steam (native version) and it consumes around 800megabytes of memory when launched, also in game the memory usage is a lot less now than around 2gigabyes what it was before.

So the loading speed has increased a lot.

@vhda
@deri82

The memory usage is so much lower and loading speed so much faster that I am not going to remove the shader preload setting. I am currently searching game.

UPDATE

I picked centaur and I am waiting game to start. Memory usage is around 1.5 gigabytes (compared 2gigabytes in game)

Update

I am in game area now, memory usage 1.6 gigabytes.

@LiranV

With "+mat_autoload_glshaders 0" and "-novid" the game now starts in 18 seconds (SSD).
This is a huge improvement over the previous 1Min+ it took.

@deri82

LiranV have you also deleted your steam's appcache directory? Removing that boosts speed. I had it grown too large.

@LiranV

I did reinstalled steam a few days ago so it's a yes :)

@felipec

Looks like not only somebody is working on this, but the fix is coming:

http://www.phoronix.com/scan.php?page=news_item&px=MTY4MjA

@vrodic

@felipec This is very nice work, but it fixes the symptoms, not the cause of problems. Dota 2 for Linux, probably because of how ToGL implements shader translation needs 2x more memory than the Windows counterpart.

I did some testing here, and the situation is still same as it was back in august of last year. http://vrodic.blogspot.com/2013/08/dota-2-performance-linuxnative-vs.html

@Weegee

I doubt that we'll ever get a full OpenGL rewrite, but why is Wine so much more efficient than Valve's ToGL wrapper? Would it be possible to improve ToGL with code from the Wine wrapper?

@felipec

This is very nice work, but it fixes the symptoms, not the cause of problems.

@vrodic If I understand correctly the problem is a mismatch between Direct3D and OpenGL, so there's not much Dota 2 or ToGL can do.

I did some testing here, and the situation is still same as it was back in august of last year.

Not with Mesa, Dota 2 and ToGL patched and the corresponding drirc configuration I bet.

It will take some time before all those changes percolate.

@felipec

I doubt that we'll ever get a full OpenGL rewrite, but why is Wine so much more efficient than Valve's ToGL wrapper? Would it be possible to improve ToGL with code from the Wine wrapper?

@Weegee That is a good question.

My bet is that ToGL is still very new and needs lots of improvements that the Wine project has had lots of time to find and implement.

@vrodic

Not with Mesa, Dota 2 and ToGL patched and the corresponding drirc configuration I bet.

It will surely affect load times, but not memory usage. There are other things people are doing for Mesa to make shaders take less memory (and that is also good), but I bet that if nothing changes in the shader translation layer, Windows version under Wine will always use less memory.

@tpruzina

Well, from litle of profiling (sysprof, strace) & memory dumping (/proc/pid) I have done:
1) Path resolution calls 'stat' on each directory in the path and often fails (for example stuff with $steamdir/steamapps/... is attemped before (default) $steamdir/SteamApps/... resulting in way too many syscall -ENOENT returns (inode mismatchs).
2) Even when idle in main mainu (alt tabbed), load is pretty big on 50% of that time dota_linux spends in vdso_getcpu.
3) ammount of craps that gets accessed/mmaped is absurd (voice files for stuff that isn't even present in game):

Some of the stuff that I stated might be incorrect tho, I merely tried debugging running instance and jacking combat logs from memory directly.

vdso_getcpu is called during wait, so I guess this has something to do with __GL_YIELD, perhaps it would be worth investigating how nvidia yield options affect busy/idle performance (usleep,none,sched).

@Plagman
Owner

The new patch should help on drivers that support threaded shader compilation, which currently is just the AMD driver. If you build Mesa from source with the threaded compile patchset applied that'll also work. We're working on further improvements in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.