Fix `config.defer_build` for serialization first cases #9771

Job	Run time
test memray	5m 10s
test ubuntu / 3.7	2m 37s
test ubuntu / 3.8	2m 10s
test ubuntu / 3.9	2m 0s
test ubuntu / 3.10	2m 38s
test ubuntu / 3.11	2m 19s
test macos / 3.7	3m 41s
test macos / 3.8	3m 2s
test macos / 3.9	3m 8s
test macos / 3.10	3m 30s
test macos / 3.11	3m 45s
test windows / 3.7	3m 25s
test windows / 3.8	3m 50s
test windows / 3.9	4m 3s
test windows / 3.10	4m 43s
test windows / 3.11	5m 45s
test ubuntu / pypy3.7	8m 54s
test ubuntu / pypy3.8	8m 50s
test ubuntu / pypy3.9	7m 59s
docs-build	1m 26s
lint 3.7	1m 16s
lint 3.8	1m 9s
lint 3.9	1m 2s
lint 3.10	1m 37s
lint 3.11	1m 31s
test fastapi	2m 12s
mypy 1.0.1 / 3.10	5m 13s
mypy lockfile / 3.10	3m 41s
mypy 1.2.0 / 3.10	4m 20s
mypy 1.4.1 / 3.10	3m 33s
mypy 1.5.0 / 3.10	4m 12s
mypy lockfile / 3.7	3m 19s
mypy lockfile / 3.8	4m 1s
mypy lockfile / 3.9	3m 53s
mypy lockfile / 3.11	3m 29s
test typing-extensions 4.6.1 on Python 3.8	3m 5s
test typing-extensions 4.6.1 on Python 3.9	3m 18s
test typing-extensions 4.6.1 on Python 3.10	2m 32s
test typing-extensions 4.6.1 on Python 3.11	3m 24s
test typing-extensions main on Python 3.8	2m 58s
test typing-extensions main on Python 3.9	2m 20s
test typing-extensions main on Python 3.10	3m 14s
test typing-extensions main on Python 3.11	2m 39s
test typing-extensions 4.6.1 on Python 3.7	1m 22s
check	3s
coverage-combine	32s
release	0s
Send tweet	0s
	2h 32m 50s

Provide feedback