Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msbuild: Merge "Core" libs into single DolphinLib #9092

Merged
merged 12 commits into from Jan 28, 2021

Conversation

shuffle2
Copy link
Contributor

@shuffle2 shuffle2 commented Sep 17, 2020

For a long time, dolphin (on msbuild at least) has been building all of the "Dolphin Core" code with identical compilation settings.
This PR takes the next logical step and actually sends all source files to the compiler in a single batch. This itself is a build speedup.

Another build speedup comes from cutting out some unnecessary lib calls, since all the resulting "core" object files get packaged into a single lib file. This should especially help buildbot/machines with few threads, since otherwise the lib step acts like a bubble in the msbuild pipeline (this is a legitimate failing of msbuild!).

Another side effect is that the msbuild project dependency graph becomes almost flat, which also improves build parallelism (really, can be attributed to the same msbuild fail as the lib thing).

I'm currently investigating if it's possible for ninja to perform batched compiler invocations, and while I see people requesting/working on the feature over the years, I don't think it's actually implemented yet(?). Anyway, hopefully we can eventually do it for ninja/msvc. The batching would still be a speedup for ninja builds, but the speedup from lib reduction would probably not be as noticeable.

A prerequisite for bundling all the source files together is that they all have unique basenames. All of the commits except the final one are just accomplishing that. In most cases I think it also results in better style anyway.

I think with this PR and #9081 , the only realistic + meaningful remaining speedup for windows buildbot (without giving it better compute resources) is passing /m to msbuild (because currently link of multiple projects is serialized), but that change would have to be done in sadm.

@Rumi-Larry
Copy link

It might be a good idea to rename "VulkanContext" to "VKContext" for consistency 🤷

@shuffle2
Copy link
Contributor Author

I obey the linter

@Rumi-Larry
Copy link

@shuffle2 I assume that for the same reason, the D3D11backend has to be called just D3D?

@shuffle2
Copy link
Contributor Author

substantive/constructive comments appreciated ...

Copy link
Member

@BhaaLseN BhaaLseN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sheer number of files changed by the renaming is probably putting off a lot of reviewers (myself included, I ignored it until now because I didn't feel like it).

Do you have any numbers comparing before/after, like full rebuild time, compile/link time, binary size (if that even changes in any way), etc...?

Source/Core/DolphinNoGUI/DolphinNoGUI.vcxproj Outdated Show resolved Hide resolved
Source/Core/DolphinQt/DolphinQt.vcxproj Outdated Show resolved Hide resolved
Source/Core/WinUpdater/WinUpdater.vcxproj Outdated Show resolved Hide resolved
@shuffle2
Copy link
Contributor Author

shuffle2 commented Oct 27, 2020

It was too long ago now, I'll have to redo the perf traces and get back about exact numbers, but it cut full rebuild time for me into about half iirc (amd 3990x).

Anyone can look themselves with vcperf.

@shuffle2 shuffle2 force-pushed the vshack branch 4 times, most recently from a937011 to 99f216b Compare January 10, 2021 07:32
@shuffle2
Copy link
Contributor Author

shuffle2 commented Jan 10, 2021

Some simple perf stats of running the following for both current master and this PR:

rmdir /s/q Build Binary
msbuild Source\dolphin-emu.sln -p:Configuration=Release,Platform=x64 -m -v:q -clp:PerformanceSummary

master:

Task Performance Summary:
        1 ms  ConvertToAbsolutePath                     47 calls
        3 ms  RemoveDuplicates                          94 calls
        7 ms  AssignCulture                             47 calls
        8 ms  CreateProperty                           244 calls
        9 ms  HostTranslatePaths                       122 calls
        9 ms  CreateItem                                94 calls
       13 ms  Message                                  173 calls
       15 ms  AssignTargetPath                         329 calls
       16 ms  FindUnderPath                            235 calls
       16 ms  CheckVCToolsetVersion                     47 calls
       21 ms  GetItemHash                              122 calls
       28 ms  AssignProjectConfiguration                47 calls
       29 ms  Flatten                                  122 calls
       42 ms  Expand                                     1 calls
       53 ms  SetEnv                                   376 calls
       55 ms  ReadLinesFromFile                         95 calls
       62 ms  GetOutOfDateItems                        141 calls
       66 ms  Delete                                    94 calls
       73 ms  GenerateDesktopDeployRecipe               47 calls
       83 ms  WriteLinesToFile                         203 calls
      121 ms  RC                                         2 calls
      181 ms  Touch                                     94 calls
      260 ms  QtRunWork                                  1 calls
      330 ms  MakeDir                                  582 calls
      541 ms  CallTarget                                98 calls
      884 ms  Copy                                     2003 calls
     3452 ms  Link                                       4 calls
     3508 ms  Exec                                       1 calls
     8854 ms  msgfmt                                    28 calls
    18039 ms  LIB                                       40 calls
    262487 ms  CL                                        47 calls
    511553 ms  MSBuild                                  132 calls

PR:

Task Performance Summary:
        4 ms  CreateProperty                           244 calls
        4 ms  ConvertToAbsolutePath                     33 calls
        4 ms  CreateItem                                66 calls
        5 ms  AssignCulture                             33 calls
        7 ms  RemoveDuplicates                          66 calls
       10 ms  HostTranslatePaths                       122 calls
       10 ms  CheckVCToolsetVersion                     33 calls
       12 ms  Message                                  131 calls
       15 ms  FindUnderPath                            165 calls
       20 ms  AssignProjectConfiguration                33 calls
       25 ms  SetEnv                                   264 calls
       26 ms  GetItemHash                              122 calls
       32 ms  AssignTargetPath                         231 calls
       37 ms  ReadLinesFromFile                         67 calls
       39 ms  Flatten                                  122 calls
       41 ms  Expand                                     1 calls
       44 ms  GetOutOfDateItems                         99 calls
       80 ms  GenerateDesktopDeployRecipe               33 calls
      100 ms  WriteLinesToFile                         189 calls
      104 ms  Delete                                    66 calls
      220 ms  CallTarget                                70 calls
      237 ms  RC                                         2 calls
      248 ms  Touch                                     66 calls
      261 ms  QtRunWork                                  1 calls
      462 ms  MakeDir                                  442 calls
      907 ms  Copy                                     2003 calls
     3595 ms  Link                                       4 calls
     7133 ms  Exec                                       1 calls
    10613 ms  msgfmt                                    28 calls
    23056 ms  LIB                                       26 calls
    169739 ms  MSBuild                                   54 calls
    253276 ms  CL                                        33 calls

The MSBuild overall time is about 1/3 of master, resulting in overall clean build time of about 1/2 (CL becomes the long pole).

fwiw, the actual wall-clock times:
Complete clean build is master: 81 seconds, PR: 60 seconds.
Clean build just Source\Core\DolphinQt\DolphinQt.vcxproj: master: 60 seconds, PR: 44 seconds
Build full solution after changing 1 file in Source\Core\Common: master: 16 seconds, PR: 13 seconds

@leoetlino leoetlino merged commit 9ca24ae into dolphin-emu:master Jan 28, 2021
10 checks passed
@shuffle2 shuffle2 deleted the vshack branch January 28, 2021 00:45
Pokechu22 added a commit to Pokechu22/dolphin that referenced this pull request Feb 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
5 participants