The future of ccache's support for running on Windows

[jrosdahl]

This issue is for discussing the future of ccache's support for running on Windows for caching GCC/Clang output. It's not about building/using ccache with Visual Studio.

The present

• ccache has had some support for running on Windows for many years. (Here "running on Windows" only means executing on Windows, not support for caching output of Windows-only compilers like Visual Studio.)
• The support has however never been official. The major reason is simply that the ccache development team (@jrosdahl) has little to no personal interest in the Windows port.
• We have CI tests that verify that ccache is possible to build with a MinGW-w64 cross-compiler.
• However, there are currently no CI tests that run on Windows. This means that the state of the Windows support is largely unknown for the ccache developers. See also CI: add Appveyor script #247.
• Every once in a while, contributors make various efforts to fix or improve the Windows support, which is appreciated.
• The main problem still remains, though: there are too few people who both care about the Windows port and are expected to contribute to ccache regularly in the future.
• As long as I (@jrosdahl) am the main ccache maintainer and contributor without more people helping out, I'm forced by time and interest constraints to be restrictive in accepting changes that make it harder for me to maintain ccache. It's easy to accept new features that work now, but it's a different story to keep them working. (I've written more about this before here, although then the context was also about using ccache for the Visual Studio compiler.)
• The problem I have with maintaining Windows-specific code is that I don't run Windows and don't know much about Windows development environments, so if some Windows-related problem is flagged by CI tests it's hard and timeconsuming for me to understand and fix them. If such problems block a release, I have to disable the tests and release with known-broken Windows support. And I want to dedicate my spare time on things I find fun to do, like implementing new features, not fixing portability issues.
• If there were more people who are interested in (and committed to) the Windows port in the developer team (i.e. more than zero 🙂), things would be different.

We need to figure out a way forward to avoid wasting time and to reduce friction between people with different goals and perceptions on what ccache should support.

The future

I see the following possibilities:

1. Keep the status quo. I.e. the Windows support continues to be unofficial and releases continue to be untested. Problems will maybe be fixed by drive-by contributions, or maybe not.
   • Pros: Nothing would need to be done at the moment.
   • Cons: ccache developers may get frustrated by getting bug reports for Windows-specific things or by having to review code changes that they don't quite understand or care about. People who want to improve the Windows port may get frustrated if their fixes are not accepted. People who want to use the Windows version will get unhappy if it doesn't work.
2. Remove all Windows support.
   • Pros: This would make the life simpler for the current ccache developers.
   • Cons: People that do want to use ccache on Windows will have maintain or contribute to a fork of their own, or start using other similar tools.
3. Remove Windows native support but keep support for a POSIX compatibility layer like MSYS2. Add CI tests for the Windows case.
   • Pros: More things will "just work" without a lot of Windows special cases. ccache developers will might be less unhappy with this. Some or many Windows users will be happy.
   • Cons: The current ccache developers might be unhappy. Windows users who for some reason can't or don't want to use the MSYS2 version will be unhappy and might have to maintain their own fork(s).
   • If there were more people who are interested in (and committed to) the Windows port in the developer team, this would be a feasible approach.
4. Add full Windows native support officially. Add CI tests for the Windows case(s). Make sure that all releases work well on Windows as well. Maybe publish official Windows builds.
   • Pros: Windows users will be happy.
   • Cons: The current ccache developers will be unhappy and might lose interest in ccache more or less completely.
   • If there were more people who are interested in (and committed to) the Windows port in the developer team, this might be a feasible approach.
5. Like 3 or 4 but create a separate sibling ccache-for-windows project. I've suggested this before (here and here).
   • To clarify, the project would:
     • Be an official sibling project and could be hosted on e.g. https://github.com/ccache/winccache or some other name.
     • Be a downstream ccache project in the sense that it would incorporate new ccache changes regularly and make sure they work fine with supported compilers and ecosystems.
     • Be a joint effort by people who want to work on and maintain Windows support. Those people can of course also be part of the main ccache project.
     • Contribute changes back to the upstream ccache project to make the life easier in the downstream project.
     • Make releases (maybe also binary releases) that match the main ccache versions. That is, when ccache X.Y is released, the winccache project also releases ccache X.Y, but not necessarily at the same time.
   • In other words: This project could be where people who want to develop ccache Windows support can collaborate instead of running their own race in different forks.
   • Pros: ccache developers will likely be happy. Windows users will be happy.
   • Cons: We need to find people who are willing to drive such an effort.

Do you see other possibilities? Do you want to help out? Please share your thoughts.

[jrosdahl]

Here is a list of people who have shown interest in ccache Windows support previously:

@AmineKhaldi
@TheOneRing
@cristianadam
@havardgraff
@ivaigult
@jmartens
@kaidokert
@leanid
@luzpaz
@melven
@nagayasu-shinya
@oded-ist
@orgads
@peterbud
@petrochenkov
@rcdailey
@rkitover
@zmike

Perhaps some of you would like to help out with maintaining or discussing ccache's support for running on Windows?

[rkitover]

Hi, I am happy to help out with this. And I've done some windows portability stuff recently. I will add some notes:

• the msys repos have both a native and msys ccache, the native ccache is not usable last I tried but the msys ccache is very helpful and works fine. The native build may be suffering from some path related issues, but it should not be too difficult to make it work.

• there is no reason we can't be more ambitious and also support visual studio. Visual studio has almost perfect cmake support, and a ccache port could be added to vcpkg. cmake itself has support for ccache. As for the visual studio compilers, they are not so different to make this too difficult, they are also a program that takes input and switches and produces an output file. On the CI side, AppVeyor can do visual studio stuff.

• in the above case, the autoconf code would need to be converted to cmake. cmake is better anyway, and the conversion process is straightforward and I've done such before.

[jrosdahl]

Hi, I am happy to help out with this.

Good to hear. Let's see if more people are interested as well.

• there is no reason we can't be more ambitious and also support visual studio. [...] they are not so different to make this too difficult

Sure, there are no technical reasons why ccache can't support Visual Studio, only organizational reasons. By that I mean that the project complexity will increase, which means that more developer time and effort will be needed.

On the CI side, AppVeyor can do visual studio stuff.

(As mentioned in the description, see also #247.)

• in the above case, the autoconf code would need to be converted to cmake. cmake is better anyway, and the conversion process is straightforward and I've done such before.

See also #430.

[peterbud]

Hi @jrosdahl : pls also count me in.

[cristianadam]

I would like some official Windows support. If not, my fork works fine with native MinGW-w64 (no MSYS2) on Windows. Since I already spent some cycles on this, I can spend some more :)

[melven]

I'm interested in Windows support as this would help our group to reduce (re)compile times significantly.
Unfortunately, I cannot afford to spend much time on this -- I could try / test it on Windows (mingw-w64) and setup CI jobs.
(the other annoying thing is linking time on Windows with large template libraries -- I'm still waiting for a usable LLVM lld as drop-in replacement for the mingw ld)

[orgads]

The latest version of lld in msys2/mingw works for me. I backported several patches for some bugs that I reported and were fixed upstream, and it now works for most cases. Did you try it recently?

[jrosdahl]

@peterbud, @cristianadam & @melven: Great, thanks.

Lately I've spent most of my spare cycles on finishing up the first steps of C++-ification, so I haven't thought much about the Windows story for a while. What do you all think is the next step? Would you like to aim for option 5 (sibling ccache-for-windows project) or option 4 (full Windows native support officially). I take it you're not interested in option 3.

What I would like to see is for official Windows support regardless of the project model:

• One or more persons who feel responsible for maintaining the Windows port. E.g.:
  • Monitor, triage and/or fix Windows-specific bugs and answer questions in GitHub issues, the gitter channel and the mailing list.
  • Help me and others with making new features work well on Windows.
• Some kind of plan in written form for what steps need to be taken. For instance: What is left to do to make native non-msys2 ccache work well? Which CI setups are needed? Is the documention good enough for the Windows case? Should the project produce prebuilt Windows binaries, and if so how and in which forms? Which changes are needed for the main web site (related: Not clear which compilers are supported #355)?

Regarding Windows-specific bugs: #450 is a good example where I need help. Any takers?

[cristianadam]

I've blogged about speeding up C++ GitHub Actions recipes using ccache, on all platforms, including Windows with Visual C++: https://cristianadam.eu/20200113/speeding-up-c-plus-plus-github-actions-using-ccache/

I've used this not only in my HelloWorld project, but also building Qt Creator which went from building in 1 hour to building in 20 minutes.

[jrosdahl]

Since I haven't received any reply to my previous comment I've been thinking some more about this.

I fully understand that it's quite hard to commit to driving and maintaining the Windows ccache port. At the same time several people do want to help out, so Windows support will probably continue to be OK enough for people to actually use it, just potentially broken and underdocumented from time to time. I've previously been thinking that the status quo is not good enough, but maybe it's actually the best option if it's just made clear that some platforms where ccache likely works in practise do not necessarily have the same level of support or attention as others.

So I've decided to go with kind of option 1b for now: document which platforms/compilers have full support and which are more on the "likely works" level. And I hope that @peterbud, @cristianadam, @melven and others who are interested in ccache for Windows continue to contribute. Keep an eye on the windows label. 🙂

[jrosdahl]

document which platforms/compilers have full support and which are more on the "likely works" level

Published now: https://ccache.dev/platform-compiler-language-support.html

[AlexanderLanin]

Even in case of option 1/1b there should probably be a (slow!!) conversion of tests from shell scripts to e.g. python, cpp or whatever. For example for newly written tests.
Otherwise this will block the windows port forever.

[llunak]

Is there still anything to do or decide here? Both Windows and MSVC support can be solved by marking them as 'B" support at https://ccache.dev/platform-compiler-language-support.html and then they are at the same level like Objective-C, other Unix-like platforms or Intel C++ compiler. I don't why Windows/MSVC should be fundamentally any different there.

The way I see it, the Windows/MSVC problem in ccache is a chicken-and-egg problem. You'd like to have people working on Windows/MSVC support, but you're pretty unlikely to get that as long as there's no Windows/MSVC support in ccache. Working on somebody else's random fork is making it more difficult, working based on pending PR's is making it more difficult, etc., and then it turns into things like #162 and #506 , which is further demotivating. FWIW, seeing all these open issues and PR's about it made me think that it must be pretty hard, and I was really surprised to see how simple #506 was. Had I known, I would have possibly written something like that sooner.

Windows/MSVC support is usable with #506 and #954, I've self-built ccache using it, and then LibreOffice. It's nowhere near the level of A features, but if you accept these and mark Windows/MSVC as B, it can get fixed with follow-up commits, and you're much more likely get more contributions when it will be as simple as working on something that is in ccache itself and already works.

As for maintainers, I think ccache has the problem that its users tend to usually have enough work already with whatever project they already use ccache for. But if there's something you don't know how to sort out, you can CC the contributors (I can count at least 3 of us) and hopefully somebody does something with it. And if not, well, B level is B level. Not to mention, if people actually start using ccache for Windows/MSVC, then if something breaks, people will be more likely to fix the problems once it's already in use.

[jrosdahl]

@llunak wrote:

You'd like to have people working on Windows/MSVC support

This sounds a bit backwards. If other people want Windows/MSVC support then ideally those other people should work on and maintain it and the complexity it brings. I'm not personally interested in Windows/MSVC support. What I am interested in, though, is being helpful, but only to the extent that I don't burn out. The challenge is finding the right balance.

I was really surprised to see how simple #506 was.

I too was surprised that MSVC support would require few changes (leading to #506 (review)). Again, the main issue is that I want to be sure that I don't accept stuff that makes the project too hard or boring for me to work with so that I don't want to do it anymore. Then everybody loses, unless somebody with more time and energy than I have wants to take over. Most bullets in The present part of #447 (comment) are still applicable.

[rkitover]

@jrosdahl Hi, hope you're doing well.

I have been working on a Windows developer setup/usage guide for Linux users on and off for the past couple of years and more lately. If you have any interest in this at all, it's here:

https://github.com/rkitover/windows-dev-guide

[Predelnik]

Personally I'm very happy with using ccache with msvc (at least after applying #992) locally. Before I was very concerned when switching branches &c., now it's no problem at all. The only issue is that msvc benefits a lot in terms of compilation speed from precompiled headers and I turn them off for now because I'm not sure how they interact with ccache currently but it's just a trade-off in the end and not a deal-breaker.

[rkitover]

I did some unscientific timings of builds of our project with msvc (https://github.com/visualboyadvance-m/visualboyadvance-m/) and the improvement with ccache is very minor, unlike it is under linux:

https://gist.github.com/rkitover/d248ef0c9af9a08dfdfd2e93a2440960

I need to see why that is, if the problem is in our project or in ccache or what. But it does work now at least.

[Predelnik]

If it's 100% hit, speed-up should be significant, it would certainly be interesting to investigate.

[Trass3r]

ccache -s -v ?

[Zitrax]

I am seeing the same thing, ccache under windows is way slower than on Linux. Was any more investigation done on that?