Support Python 3.12

[justinvp]

Python 3.12 requires grpcio 1.59.0 or higher. Unfortunately, there is a regression in grpcio 1.58.0 through the latest version (currently 1.60.0) which causes any error returned from a Python gRPC server to be written to stderr, including UNIMPLEMENTED errors. This primarily affects Python dynamic providers, which don't have implementations for CheckConfig and DiffConfig, resulting in a traceback error being emitted to stderr when the engine calls these, which is visible to users. This grpcio regression has been fixed upstream, but the fix has not been released yet. We've been waiting for a 1.60.1 patch release.

This has not been great for our Python users who are using Python 3.12. It's particularly bad for new Pulumi users who are using Python 3.12 and are trying to get started with Pulumi. For these users, when trying to install the pulumi PyPi package (i.e. via pulumi new python) the installation fails with an error because it is pinned to depending on an older version of grpcio which doesn't work on Python 3.12.

This commit works around the problem by providing default implementations of CheckConfig and DiffConfig for python dynamic providers and the component provider API, so that no error is emitted to stderr when the engine calls these methods. The default implementations for these are the same behavior that the engine would use if these methods had returned UNIMPLEMENTED. I believe these are the only two methods affected by this. Other methods like Invoke, Call, StreamInvoke, Construct, Attach, GetMapping, and GetMappings, continue to return UNIMPLEMENTED for dynamic providers, which I think is OK; I don't believe these will be called by the engine under normal circumstances.

Out of an abundance of caution, the pulumi package continues to depend on the pinned version of grpcio when installing on versions of Python less than 3.12. On Python 3.12 or greater, we now depend on grpcio ~=1.60.0. 1.60.0 doesn't have the fix for the regression, but the workaround should allow things to work on Python 3.12 as before.

Once 1.60.1 is released, we can look into updating the grpcio dependency to ~=1.60.1 for all versions of Python, and possibly revert the workarounds, if we want.

Note: #14474 added a test for dynamic providers to ensure nothing is written to stderr. The test would fail if the workaround in this PR did not work as intended: https://github.com/pulumi/pulumi/pull/14474/files#diff-d92ccd283e08eadab2597825103e45cdaa96fea93324bc4d4d3b1d2b83c51b76

This PR depends on several other smaller PRs:

• [ci] Install setuptools in virtual environment #15220
• Upgrade pylint to 3.0.3 #15221
• [sdk/python] Install local SDK in automation API test #15222
• [sdk/python] Fix test to install local Pulumi SDK #15223
• [tests/integration] Fix resource_args for Python 3.12 #15224
• ProgramTest: Python: Use correct venv path for projects with main option #15225
• [tests/integration] Remove pulumi from requirements.txt #15226

Fixes #14258

[pulumi-bot]

Changelog

[uncommitted] (2024-01-24)

Features

• [sdk/python] Add support for Python 3.12
  #15190

[justinvp]

This matches the behavior of the engine if this is UNIMPLEMENTED:

pulumi/sdk/go/common/resource/plugin/provider_plugin.go

Lines 326 to 330 in 8bcef51

	if code == codes.Unimplemented || isDiffCheckConfigLogicallyUnimplemented(rpcError, urn.Type()) {
	// For backwards compatibility, just return the news as if the provider was okay with them.
	logging.V(7).Infof("%s unimplemented rpc: returning news as is", label)
	return news, nil, nil
	}

[justinvp]

This matches the behavior of the engine if this is UNIMPLEMENTED:

pulumi/sdk/go/common/resource/plugin/provider_plugin.go

Lines 439 to 454 in 8bcef51

	if code == codes.Unimplemented || isDiffCheckConfigLogicallyUnimplemented(rpcError, urn.Type()) {
	logging.V(7).Infof("%s unimplemented rpc: returning DiffUnknown with no replaces", label)
	// In this case, the provider plugin did not implement this and we have to provide some answer:
	//
	// There are two interesting scenarios with the present gRPC interface:
	// 1. Configuration differences in which all properties are known
	// 2. Configuration differences in which some new property is unknown.
	//
	// In both cases, we return a diff result that indicates that the provider _should not_ be replaced.
	// Although this decision is not conservative--indeed, the conservative decision would be to always require
	// replacement of a provider if any input has changed--we believe that it results in the best possible user
	// experience for providers that do not implement DiffConfig functionality. If we took the conservative
	// route here, any change to a provider's configuration (no matter how inconsequential) would cause all of
	// its resources to be replaced. This is clearly a bad experience, and differs from how things worked prior
	// to first-class providers.
	return DiffResult{Changes: DiffUnknown, ReplaceKeys: nil}, nil

[justinvp]

Looks like there are some issues when running CI with Python 3.12:

• pylint issue
• Issue with setuptools no longer being included in virtual environments created with python -m venv

Edit: Fixed

[justinvp]

More test failures to investigate.

[tgummerer]

I do think we should revert the workarounds once we can update to 1.60.1, but this is a good step forward in the meantime!