[sdk/python] Support Sequence[T] for array types rather than List[T] #5282

justinvp · 2020-09-03T07:25:54Z

We currently emit array types as List[T] for Python, but List[T] is invariant, which causes type checkers like mypy to produce errors when values like ["foo", "bar"] are passed as args typed as List[pulumi.Input[str]] (since Input[str] is an alias for Union[T, Awaitable[T], Output[T]]. To address this, we should move to using Sequence[T] which is covariant, and does not have this problem.

We actually already do this for Dict vs. Mapping, emitting map types as Mapping[str, T] rather than Dict[str, T] because Mapping[str, T] is covariant for the value. This change makes us consistent for array types.

These are the SDK changes necessary to support Sequence[T].

Codegen PR: #5283

Part of #5278

justinvp · 2020-09-03T07:28:38Z

sdk/python/lib/pulumi/runtime/rpc.py

@@ -125,7 +127,7 @@ async def serialize_property(value: 'Input[Any]',
            remote_asset = cast('RemoteAsset', value)
            obj["uri"] = await serialize_property(remote_asset.uri, deps, input_transformer)
        else:
-            raise AssertionError(f"unknown asset type: {value}")
+            raise AssertionError(f"unknown asset type: {value!r}")


Otherwise mypy errors:

error: On Python 3 '{}'.format(b'abc') produces "b'abc'"; use !r if this is a desired behavior

due to the not isinstance(value, bytes) above.

justinvp · 2020-09-03T07:36:37Z

sdk/python/lib/pulumi/runtime/rpc.py

@@ -194,7 +196,7 @@ async def serialize_property(value: 'Input[Any]',
        value = _types.input_type_to_dict(value)
        transform_keys = False

-    if isinstance(value, Mapping):  # pylint: disable=bad-option-value,isinstance-second-argument-not-valid-type
+    if isinstance(value, abc.Mapping):


While the previous code worked, we should be checking against the abstract base class from collections.abc rather than the type from typing.

justinvp · 2020-09-03T07:37:08Z

sdk/python/lib/test/test_next_serialize.py

@@ -530,7 +550,7 @@ async def test_apply_does_not_propagate_secret_on_unknown_unknown_output_during_
        settings.SETTINGS.dry_run = True

        out = self.create_output(0, is_known=False)
-        r = out.apply(lambda v: self.create_output("inner", is_known=false, is_secret=True))
+        r = out.apply(lambda v: self.create_output("inner", is_known=False, is_secret=True))


Noticed this when I had the file open in VS Code with Pylance installed.

But... how did it work before?

The lambda passed to apply is not called because settings.SETTINGS.dry_run = True. If settings.SETTINGS.dry_run was set to False, then it fails as you'd expect with:

NameError: name 'false' is not defined

justinvp · 2020-09-03T07:38:12Z

sdk/python/lib/test/test_next_serialize.py

@@ -951,7 +988,18 @@ class InputTypeSerializationTests(unittest.TestCase):
    async def test_simple_input_type(self):
        it = FooArgs(first_arg="hello", second_arg=42)
        prop = await rpc.serialize_property(it, [])
-        self.assertDictEqual(prop, {"firstArg": "hello", "secondArg": 42})
+        self.assertEqual({"firstArg": "hello", "secondArg": 42}, prop)


assertEqual will use assertDictEqual internally per https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertDictEqual

This method will be used by default to compare dictionaries in calls to assertEqual().

And for consistency elsewhere, specify the expected value first.

justinvp · 2020-09-03T17:45:09Z

This is a non-breaking change. Python doesn’t have types at runtime, so nothing would be hard broken by the change.

It is possible someone using mypy (or some other type checker) will see new warnings, depending on what they’re doing. In particular, if someone happened to be mutating a resulting list, they'll likely see new warnings from their type checker because Sequence is a readonly interface whereas List is mutable (this is a non-issue at runtime as the instance will be a mutable list).

We already did this for maps, though, and I haven't heard any feedback from users about it. Previously maps were annotated as dict (mutable), and the new type annotations recently released use Mapping[str, T] (readonly). So this change would just be doing the same for lists.

I do wonder if we should consider annotating outputs (return types) as MutableSequence[T] or List[T], and doing the same for maps, annotating outputs as MutableMapping[str, T] or Dict[str, T]. However, I'm inclined to keep these annotated as the readonly Sequence[T] and Mapping[str, T] types for both inputs and outputs for now. This aligns with what we've done in other languages (e.g. .NET uses ImmutableArray<T> and ImmutableDictionary<string, TValue>). If we hear feedback otherwise from users, we can always consider changing the annotations for results to be the mutable variants.

komalali · 2020-09-08T16:20:27Z

sdk/python/lib/test/test_next_serialize.py

@@ -530,7 +550,7 @@ async def test_apply_does_not_propagate_secret_on_unknown_unknown_output_during_
        settings.SETTINGS.dry_run = True

        out = self.create_output(0, is_known=False)
-        r = out.apply(lambda v: self.create_output("inner", is_known=false, is_secret=True))
+        r = out.apply(lambda v: self.create_output("inner", is_known=False, is_secret=True))


But... how did it work before?

We currently emit array types as `List[T]` for Python, but `List[T]` is invariant, which causes type checkers like mypy to produce errors when values like `["foo", "bar"]` are passed as args typed as `List[pulumi.Input[str]]`. Instead, we should move to using `Sequence[T]` which is covariant, and does not have this problem. We actually already do this for `Dict` vs. `Mapping`, emitting map types as `Mapping[str, T]` rather than `Dict[str, T]` because `Mapping[str, T]` is covariant. This change makes us consistent for array types.

Similar to pulumi#5282, but for core SDK types. The tl;dr is that because Sequence[T] is covariant, constructing resources becomes much more ergonomic. Fix pulumi#7693.

Similar to #5282, but for core SDK types. The tl;dr is that because Sequence[T] is covariant, constructing resources becomes much more ergonomic. Fix #7693.

justinvp commented Sep 3, 2020

View reviewed changes

justinvp mentioned this pull request Sep 3, 2020

[codegen/python] Emit array types as Sequence[T] instead of List[T] #5283

Merged

justinvp force-pushed the justin/python_sdk_seq branch from d53dcfa to 3f76946 Compare September 3, 2020 07:43

justinvp requested review from komalali and pgavlin September 3, 2020 07:44

komalali approved these changes Sep 8, 2020

View reviewed changes

justinvp force-pushed the justin/python_sdk_seq branch from 3f76946 to cb63344 Compare September 9, 2020 04:14

justinvp merged commit d0ba9fb into master Sep 9, 2020

pulumi-bot deleted the justin/python_sdk_seq branch September 9, 2020 05:22

benesch mentioned this pull request Aug 3, 2021

python: Use Sequence rather than List in Resource fields #7700

Merged

3 tasks

Frassle mentioned this pull request Jun 20, 2023

Allow tuples as sequences in Python serialization #13210

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sdk/python] Support Sequence[T] for array types rather than List[T] #5282

[sdk/python] Support Sequence[T] for array types rather than List[T] #5282

justinvp commented Sep 3, 2020 •

edited

Loading

justinvp Sep 3, 2020

justinvp Sep 3, 2020

justinvp Sep 3, 2020

komalali Sep 8, 2020

justinvp Sep 8, 2020

justinvp Sep 3, 2020 •

edited

Loading

justinvp commented Sep 3, 2020 •

edited

Loading

komalali Sep 8, 2020

[sdk/python] Support Sequence[T] for array types rather than List[T] #5282

[sdk/python] Support Sequence[T] for array types rather than List[T] #5282

Conversation

justinvp commented Sep 3, 2020 • edited Loading

justinvp Sep 3, 2020

Choose a reason for hiding this comment

justinvp Sep 3, 2020

Choose a reason for hiding this comment

justinvp Sep 3, 2020

Choose a reason for hiding this comment

komalali Sep 8, 2020

Choose a reason for hiding this comment

justinvp Sep 8, 2020

Choose a reason for hiding this comment

justinvp Sep 3, 2020 • edited Loading

Choose a reason for hiding this comment

justinvp commented Sep 3, 2020 • edited Loading

komalali Sep 8, 2020

Choose a reason for hiding this comment

justinvp commented Sep 3, 2020 •

edited

Loading

justinvp Sep 3, 2020 •

edited

Loading

justinvp commented Sep 3, 2020 •

edited

Loading