Restore caching of externals on Windows buildbots

[jkloth]

BPO	46789
Nosy	@pfmoore, @vstinner, @tjguk, @jkloth, @zware, @zooba, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-18.17:00:25.553>
labels = ['build', 'tests', 'OS-windows']
title = 'Restore caching of externals on Windows buildbots'
updated_at = <Date 2022-02-21.21:00:00.669>
user = 'https://github.com/jkloth'

bugs.python.org fields:

activity = <Date 2022-02-21.21:00:00.669>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'Tests', 'Windows']
creation = <Date 2022-02-18.17:00:25.553>
creator = 'jkloth'
dependencies = []
files = []
hgrepos = []
issue_num = 46789
keywords = []
message_count = 5.0
messages = ['413494', '413498', '413650', '413655', '413680']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'jkloth', 'jeremy.kloth', 'zach.ware', 'steve.dower', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46789'
versions = []

[jkloth]

A recent change to the buildmaster config effectively disabled the caching of the externals for Windows buildbots:
python/buildmaster-config#255

If the caching is desired, a simple change to the buildmaster config is needed (define EXTERNALS_DIR in the build environment). Or, to continue with fetching them each run, the buildbot scripts in Tools\buildbot can be simplified.

Once a course of action is determined I can develop the requisite PR(s) in the appropriate tracker.

[jkloth]

I personally would like to see caching restored so as to keep the duration of buildbot runs as low as possible. The repeated fetching effectively doubles compilation time for my Win11 builder.

[vstinner]

Would it be possible to create a download cache somewhere outside the Python source tree, so "git clean -fdx" would not remove this cache?

Some CIs implement such cache. Does buildbot have helpers for that?

buildbot provides for example:

• LRUCache: https://docs.buildbot.net/2.4.0/full.html#module-buildbot.util.lru
• State: https://docs.buildbot.net/2.4.0/full.html#keeping-state
• Server cache: https://docs.buildbot.net/2.4.0/full.html#caches

In this case, one option would be to enhance PCbuild/get_external.py to add support for a cache directory.

[jeremykloth]

Would it be possible to create a download cache somewhere outside the Python source tree, so "git clean -fdx" would not remove this cache?

I was thinking of locating it next to the checkout directory. The
current structure is:

[worker root]
-- [builder root]
---- [checkout]

I propose to add the externals directory within the builder root, so
each branch would still have a unique copy.

[zooba]

one option would be to enhance PCbuild/get_external.py to add support
for a cache directory.

It should already have this support - set the EXTERNALS_DIR environment
variable before building. I use this in my own builds.

Though if there are specific platforms worth integrating with, this
could no doubt be further improved.