Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Use buildcache also for msbuild #3186

Closed
wants to merge 1 commit into from

Conversation

@orgads
Copy link
Contributor

@orgads orgads commented Jul 21, 2021

  • Buildcache doesn't work with PDB, so switch to Release build.
  • Object file must be explicit, so adapt create_project. Set UseMultiToolTask to maintain parallel build.
@orgads orgads requested a review from SupSuper Jul 21, 2021
Copy link
Contributor

@SupSuper SupSuper left a comment

  • Buildcache doesn't work with PDB, so switch to Release build.

GitHub VMs don't have enough disk space for Release builds:

2021-07-21T21:51:27.0375767Z ##[warning]You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 99 MB
2021-07-21T21:53:57.1912338Z      2>D:\a\scummvm\scummvm\video\flic_decoder.cpp : fatal error C1085: Cannot write compiler generated file: 'D:\a\scummvm\scummvm\build-scummvm\Releasex64\scummvm\dists\video\flic_decoder.obj': No space left on device [D:\a\scummvm\scummvm\build-scummvm\scummvm.vcxproj]

They're also much slower.

  • Object file must be explicit, so adapt create_project.

This will also slow down builds dramatically, as it will force every source file to be compiled individually. cl.exe can only batch files that share compiler flags, eg:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\bin\HostX64\x64\CL.exe /c /I. /I.. /I..\engines /Zi /nologo /W4 /WX- /diagnostics:column /MP /fsanitize=address /Od /D WIN32 /D USE_ZLIB /D USE_MAD /D USE_OGG /D USE_VORBIS /D USE_FLAC /D USE_PNG /D USE_FAAD /D USE_MPEG2 /D USE_THEORADEC /D USE_FREETYPE2 /D USE_JPEG /D USE_FLUIDSYNTH /D USE_LIBCURL /D USE_SDL_NET /D USE_DISCORD /D USE_GLEW /D USE_BINK /D USE_SCALERS /D USE_HQ_SCALERS /D USE_EDGE_SCALERS /D USE_ASPECT /D USE_RGB_COLOR /D USE_HIGHRES /D USE_MT32EMU /D USE_LUA /D USE_TINYGL /D USE_OPENGL /D USE_OPENGL_GAME /D USE_OPENGL_SHADERS /D USE_TASKBAR /D USE_CLOUD /D USE_TRANSLATION /D USE_SYSDIALOGS /D USE_DETECTLANG /D USE_TTS /D BUILTIN_RESOURCES /D USE_DETECTION_FEATURES_STATIC /D USE_CXX11 /D ENABLE_SCUMM /D ENABLE_SCUMM_7_8 /D ENABLE_HE /D ENABLE_ACCESS /D ENABLE_ADL /D ENABLE_AGI /D ENABLE_AGOS /D ENABLE_AGOS2 /D ENABLE_AGS /D ENABLE_ASYLUM /D ENABLE_AVALANCHE /D ENABLE_BBVS /D ENABLE_BLADERUNNER /D ENABLE_BURIED /D ENABLE_CGE /D ENABLE_CGE2 /D ENABLE_CHEWY /D ENABLE_CINE /D ENABLE_COMPOSER /D ENABLE_CRUISE /D ENABLE_CRYO /D ENABLE_CRYOMNI3D /D ENABLE_VERSAILLES /D ENABLE_DIRECTOR /D ENABLE_DM /D ENABLE_DRACI /D ENABLE_DRAGONS /D ENABLE_DRASCULA /D ENABLE_DREAMWEB /D ENABLE_GLK /D ENABLE_GNAP /D ENABLE_GOB /D ENABLE_GRIFFON /D ENABLE_GRIM /D ENABLE_MONKEY4 /D ENABLE_GROOVIE /D ENABLE_GROOVIE2 /D ENABLE_HADESCH /D ENABLE_HDB /D ENABLE_HOPKINS /D ENABLE_HUGO /D ENABLE_ICB /D ENABLE_ILLUSIONS /D ENABLE_KINGDOM /D ENABLE_KYRA /D ENABLE_LOL /D ENABLE_EOB /D ENABLE_LAB /D ENABLE_LASTEXPRESS /D ENABLE_LILLIPUT /D ENABLE_LURE /D ENABLE_MACVENTURE /D ENABLE_MADE /D ENABLE_MADS /D ENABLE_MADSV2 /D ENABLE_MOHAWK /D ENABLE_CSTIME /D ENABLE_RIVEN /D ENABLE_MYST /D ENABLE_MYSTME /D ENABLE_MORTEVIELLE /D ENABLE_MUTATIONOFJB /D ENABLE_MYST3 /D ENABLE_NANCY /D ENABLE_NEVERHOOD /D ENABLE_NGI /D ENABLE_PARALLACTION /D ENABLE_PEGASUS /D ENABLE_PETKA /D ENABLE_PINK /D ENABLE_PLUMBERS /D ENABLE_PRINCE /D ENABLE_PRIVATE /D ENABLE_QUEEN /D ENABLE_SAGA /D ENABLE_IHNM /D ENABLE_SAGA2 /D ENABLE_SCI /D ENABLE_SCI32 /D ENABLE_SHERLOCK /D ENABLE_SKY /D ENABLE_SLUDGE /D ENABLE_STARK /D ENABLE_STARTREK /D ENABLE_SUPERNOVA /D ENABLE_SWORD1 /D ENABLE_SWORD2 /D ENABLE_SWORD25 /D ENABLE_TEENAGENT /D ENABLE_TESTBED /D ENABLE_TINSEL /D ENABLE_TITANIC /D ENABLE_TOLTECS /D ENABLE_TONY /D ENABLE_TOON /D ENABLE_TOUCHE /D ENABLE_TRECISION /D ENABLE_TSAGE /D ENABLE_TUCKER /D ENABLE_TWINE /D ENABLE_ULTIMA /D ENABLE_VOYEUR /D ENABLE_WAGE /D ENABLE_WINTERMUTE /D ENABLE_WME3D /D ENABLE_FOXTAIL /D ENABLE_HEROCRAFT /D ENABLE_XEEN /D ENABLE_ZVISION /D WIN32 /D CURL_STATICLIB /D SDL_BACKEND /D USE_SDL2 /D DETECTION_STATIC /D _UNICODE /D UNICODE /Gm- /RTC1 /MDd /GS /Gy /fp:precise /Za /permissive- /Zc:wchar_t /Zc:forScope /Zc:inline /GR /Fo"Analysisx64\ags\dists\msvc\..\engines\ags\engine\ac\\" /Fd"Analysisx64\ags.pdb" /external:env:EXTERNAL_INCLUDE /external:W4 /Gd /TP /wd4068 /wd4100 /wd4103 /wd4127 /wd4244 /wd4250 /wd4310 /wd4345 /wd4351 /wd4512 /wd4589 /wd4702 /wd4706 /wd4800 /wd4996 /wd6204 /wd6211 /wd6385 /wd6386 /wd4267 /wd4577 /analyze /FC /errorReport:queue /utf-8 ..\engines\ags\engine\ac\audio_channel.cpp ..\engines\ags\engine\ac\audio_clip.cpp ..\engines\ags\engine\ac\button.cpp ..\engines\ags\engine\ac\cd_audio.cpp ..\engines\ags\engine\ac\character.cpp ..\engines\ags\engine\ac\character_extras.cpp ..\engines\ags\engine\ac\character_info_engine.cpp ..\engines\ags\engine\ac\date_time.cpp ..\engines\ags\engine\ac\dialog.cpp ..\engines\ags\engine\ac\dialog_options_rendering.cpp ..\engines\ags\engine\ac\display.cpp ..\engines\ags\engine\ac\draw.cpp ..\engines\ags\engine\ac\draw_software.cpp ..\engines\ags\engine\ac\drawing_surface.cpp ..\engines\ags\engine\ac\dynamic_sprite.cpp ..\engines\ags\engine\ac\event.cpp ..\engines\ags\engine\ac\file.cpp ..\engines\ags\engine\ac\game.cpp ..\engines\ags\engine\ac\game_setup.cpp ..\engines\ags\engine\ac\game_state.cpp ..\engines\ags\engine\ac\global_api.cpp ..\engines\ags\engine\ac\global_audio.cpp ..\engines\ags\engine\ac\global_button.cpp ..\engines\ags\engine\ac\global_character.cpp ..\engines\ags\engine\ac\global_date_time.cpp ..\engines\ags\engine\ac\global_debug.cpp ..\engines\ags\engine\ac\global_dialog.cpp ..\engines\ags\engine\ac\global_display.cpp ..\engines\ags\engine\ac\global_drawing_surface.cpp ..\engines\ags\engine\ac\global_dynamic_sprite.cpp ..\engines\ags\engine\ac\global_file.cpp ..\engines\ags\engine\ac\global_game.cpp ..\engines\ags\engine\ac\global_gui.cpp ..\engines\ags\engine\ac\global_hotspot.cpp ..\engines\ags\engine\ac\global_inv_window.cpp ..\engines\ags\engine\ac\global_inventory_item.cpp ..\engines\ags\engine\ac\global_label.cpp ..\engines\ags\engine\ac\global_listbox.cpp ..\engines\ags\engine\ac\global_mouse.cpp ..\engines\ags\engine\ac\global_object.cpp ..\engines\ags\engine\ac\global_overlay.cpp ..\engines\ags\engine\ac\global_palette.cpp ..\engines\ags\engine\ac\global_parser.cpp ..\engines\ags\engine\ac\global_record.cpp ..\engines\ags\engine\ac\global_region.cpp ..\engines\ags\engine\ac\global_room.cpp ..\engines\ags\engine\ac\global_screen.cpp ..\engines\ags\engine\ac\global_slider.cpp ..\engines\ags\engine\ac\global_string.cpp ..\engines\ags\engine\ac\global_textbox.cpp ..\engines\ags\engine\ac\global_timer.cpp ..\engines\ags\engine\ac\global_translation.cpp ..\engines\ags\engine\ac\global_video.cpp ..\engines\ags\engine\ac\global_view_frame.cpp ..\engines\ags\engine\ac\global_viewport.cpp ..\engines\ags\engine\ac\global_walk_behind.cpp ..\engines\ags\engine\ac\global_walkable_area.cpp ..\engines\ags\engine\ac\gui.cpp ..\engines\ags\engine\ac\gui_control.cpp ..\engines\ags\engine\ac\gui_inv.cpp ..\engines\ags\engine\ac\hotspot.cpp ..\engines\ags\engine\ac\interface_button.cpp ..\engines\ags\engine\ac\interface_element.cpp ..\engines\ags\engine\ac\inv_window.cpp ..\engines\ags\engine\ac\inventory_item.cpp ..\engines\ags\engine\ac\label.cpp ..\engines\ags\engine\ac\listbox.cpp ..\engines\ags\engine\ac\math.cpp ..\engines\ags\engine\ac\mouse.cpp ..\engines\ags\engine\ac\move_list.cpp ..\engines\ags\engine\ac\object.cpp ..\engines\ags\engine\ac\overlay.cpp ..\engines\ags\engine\ac\parser.cpp ..\engines\ags\engine\ac\properties.cpp ..\engines\ags\engine\ac\region.cpp ..\engines\ags\engine\ac\rich_game_media.cpp ..\engines\ags\engine\ac\room.cpp ..\engines\ags\engine\ac\room_object.cpp ..\engines\ags\engine\ac\room_status.cpp ..\engines\ags\engine\ac\route_finder.cpp ..\engines\ags\engine\ac\route_finder_impl.cpp ..\engines\ags\engine\ac\route_finder_impl_legacy.cpp ..\engines\ags\engine\ac\route_finder_jps.cpp ..\engines\ags\engine\ac\screen.cpp ..\engines\ags\engine\ac\screen_overlay.cpp ..\engines\ags\engine\ac\script_containers.cpp ..\engines\ags\engine\ac\slider.cpp ..\engines\ags\engine\ac\speech.cpp ..\engines\ags\engine\ac\sprite.cpp ..\engines\ags\engine\ac\sprite_cache_engine.cpp ..\engines\ags\engine\ac\string.cpp ..\engines\ags\engine\ac\sys_events.cpp ..\engines\ags\engine\ac\system.cpp ..\engines\ags\engine\ac\textbox.cpp ..\engines\ags\engine\ac\timer.cpp ..\engines\ags\engine\ac\translation.cpp ..\engines\ags\engine\ac\view_frame.cpp ..\engines\ags\engine\ac\viewport_script.cpp ..\engines\ags\engine\ac\walk_behind.cpp ..\engines\ags\engine\ac\walkable_area.cpp
@orgads orgads force-pushed the orgads:buildcache-windows branch 2 times, most recently from 9f6ef90 to 4df8de3 Jul 22, 2021
@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 22, 2021

  • Buildcache doesn't work with PDB, so switch to Release build.

GitHub VMs don't have enough disk space for Release builds:

2021-07-21T21:51:27.0375767Z ##[warning]You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 99 MB
2021-07-21T21:53:57.1912338Z      2>D:\a\scummvm\scummvm\video\flic_decoder.cpp : fatal error C1085: Cannot write compiler generated file: 'D:\a\scummvm\scummvm\build-scummvm\Releasex64\scummvm\dists\video\flic_decoder.obj': No space left on device [D:\a\scummvm\scummvm\build-scummvm\scummvm.vcxproj]

Actually release build should be smaller than debug, because it doesn't produce PDB files. The space issue is probably because of the cache. It reached 2G. I configured it now to use hard links, let's see how it goes.

They're also much slower.

  • Object file must be explicit, so adapt create_project.

This will also slow down builds dramatically, as it will force every source file to be compiled individually. cl.exe can only batch files that share compiler flags, eg:

  1. Should be better now, I set UseMultiToolTask in the project.
  2. Even if it is slower, the cache should compensate on that.
@orgads orgads force-pushed the orgads:buildcache-windows branch from 4df8de3 to d47fe21 Jul 22, 2021
@orgads orgads requested a review from SupSuper Jul 22, 2021
@orgads orgads force-pushed the orgads:buildcache-windows branch 5 times, most recently from f8ec22a to 24520ca Jul 22, 2021
@mgerhardy
Copy link
Contributor

@mgerhardy mgerhardy commented Jul 22, 2021

Another option is to try to support unity builds for cmake [1]. That should drastically speed up compilation - but I'm not sure how well this plays with the build cache. But imo worth a test.

@orgads orgads force-pushed the orgads:buildcache-windows branch 2 times, most recently from fca1286 to 528a690 Jul 22, 2021
@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 22, 2021

Another option is to try to support unity builds for cmake [1]. That should drastically speed up compilation - but I'm not sure how well this plays with the build cache. But imo worth a test.

We don't use cmake, and even if we would - I doubt unity build speedup would be similar to caching.

@orgads orgads force-pushed the orgads:buildcache-windows branch 6 times, most recently from dc5fc76 to 8c32a0e Jul 22, 2021
* Buildcache doesn't work with PDB, so switch to Release build.
* Object file must be explicit, so adapt create_project. Set UseMultiToolTask
  to maintain parallel build.
@orgads orgads force-pushed the orgads:buildcache-windows branch from 8c32a0e to f209464 Jul 22, 2021
@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 22, 2021

Ok, I moved the build directory to C: by using a symbolic link. C: has 87G, so there should be no space problem again.

The build is now actually faster. It took ~45 minutes which is 50% less than usual builds. And that's without any cache hits.

I'll separate the debug->release part to see if this speeds things up.

Regarding the cache, we're beyond the 5G limit unfortunately. Each Windows is ~1.7G, macOS and ios are 1.1G and 1.4G, and each Linux is ~750M. The total is 9G. And I didn't count the vcpkg cache :/

Maybe we can use an external storage for the cache? What do you say?

@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 22, 2021

hmmmmm... It looks like the cache doesn't help much. This build has 99% hit rate, but still the build took ~38min. I wonder why..

@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 23, 2021

Ongoing discussion: mbitsnbites/buildcache#225

No reason to merge at this point. Leaving open in case we succeed (but even then - the cache size is too large...)

@SupSuper
Copy link
Contributor

@SupSuper SupSuper commented Jul 23, 2021

  1. Should be better now, I set UseMultiToolTask in the project.
  2. Even if it is slower, the cache should compensate on that.

Any changes you apply to create_project will affect all developers, so anything only useful for CI should be gated by a feature flag. We don't have cache systems on our local machines.

I think the only way you can cache MSVC builds "effectively" is to replace msbuild with something like CMake/Ninja (which still use cl.exe), and that is much more involved.

@orgads
Copy link
Contributor Author

@orgads orgads commented Jul 24, 2021

  1. I'm not sure if passing multiple source files to cl is significantly faster than invoking multiple cl processes. From what I know, Ninja executes multiple cl in parallel, and doesn't pass multiple source files. And on local machines you usually use incremental build, which is rather fast anyway.
  2. I don't see how replacing msbuild with Ninja helps wrt caching. The only advantage I see with Ninja is that we can utilize unity build, as you suggested. But like you already know, this is not an easy task at all:
    2.1. We will need to start a new cmake project from scratch, and the scummvm build logic is very complex.
    2.2. Unity build will probably require changes in the source code (for example, in case we have different static global variables with the same name).
    2.3. We will still need to maintain both the configure script and create_project for other platforms. I doubt anyone would be in favor of maintaining yet another build system.

Anyway, this discussion is currently irrelevant, since buildcache seems to have very low (or even negative) impact on build times.

@SupSuper
Copy link
Contributor

@SupSuper SupSuper commented Jul 24, 2021

I'm closing this for now then, you can reopen it if there's any updates.

@SupSuper SupSuper closed this Jul 24, 2021
@orgads orgads deleted the orgads:buildcache-windows branch Jul 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants