Codegenerate StructureShape, BlobShape, application, server and Python runtime #1403

crisidev · 2022-05-23T12:29:16Z

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Generate errors and implement error conversion. Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

github-actions · 2022-05-23T12:53:17Z

A new generated diff is ready to view.

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	10.45%	67390.31	61012.18
Total requests	10.47%	6070367	5495122
Total errors	NaN%	0	0
Total successes	10.47%	6070367	5495122
Average latency ms	26.76%	0.9	0.71
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	30.99%	21.64	16.52
Stdev latency ms	58.24%	1.44	0.91
Transfer Mb	10.47%	631.02	571.22
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

github-actions · 2022-05-23T15:10:42Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	5.06%	38569.19	36712.83
Total requests	5.11%	3474887	3305971
Total errors	NaN%	0	0
Total successes	5.11%	3474887	3305971
Average latency ms	-2.20%	0.89	0.91
Minimum latency ms	-33.33%	0.02	0.03
Maximum latency ms	-4.45%	19.13	20.02
Stdev latency ms	16.39%	0.71	0.61
Transfer Mb	5.11%	361.22	343.66
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-05-23T16:17:06Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-2.79%	73331.78	75435.49
Total requests	-2.83%	6604353	6796506
Total errors	NaN%	0	0
Total successes	-2.83%	6604353	6796506
Average latency ms	-7.04%	0.66	0.71
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	17.74%	21.04	17.87
Stdev latency ms	-14.29%	0.96	1.12
Transfer Mb	-2.83%	686.53	706.5
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-05-23T18:05:10Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	0.94%	71465.14	70796.63
Total requests	0.94%	6434729	6374545
Total errors	NaN%	0	0
Total successes	0.94%	6434729	6374545
Average latency ms	8.65%	1.13	1.04
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	45.88%	34.34	23.54
Stdev latency ms	11.70%	1.91	1.71
Transfer Mb	0.94%	668.89	662.64
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-05-23T19:40:28Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	9.18%	78921.15	72284.45
Total requests	9.19%	7110228	6511760
Total errors	NaN%	0	0
Total successes	9.19%	7110228	6511760
Average latency ms	28.38%	0.95	0.74
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	15.51%	22.27	19.28
Stdev latency ms	42.11%	1.62	1.14
Transfer Mb	9.19%	739.11	676.9
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

codegen-server-test/python/build.gradle.kts

github-actions · 2022-05-24T11:17:33Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-1.01%	85915.53	86792.88
Total requests	-0.95%	7737265	7811493
Total errors	NaN%	0	0
Total successes	-0.95%	7737265	7811493
Average latency ms	9.65%	1.25	1.14
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	-10.20%	25.43	28.32
Stdev latency ms	4.72%	2.22	2.12
Transfer Mb	-0.95%	804.29	812.01
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-05-24T11:46:54Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	2.08%	88795.1	86985.81
Total requests	2.07%	7995892	7833927
Total errors	NaN%	0	0
Total successes	2.07%	7995892	7833927
Average latency ms	-4.17%	0.46	0.48
Minimum latency ms	0.00%	0.01	0.01
Maximum latency ms	-41.43%	16.61	28.36
Stdev latency ms	-6.94%	0.67	0.72
Transfer Mb	2.07%	831.18	814.34
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

Add decorator to add `use aws_smithy_http_server_python::types` to lib.rs.

github-actions · 2022-05-24T12:13:45Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	7.38%	73412.13	68366.65
Total requests	7.30%	6609403	6159650
Total errors	NaN%	0	0
Total successes	7.30%	6609403	6159650
Average latency ms	21.65%	1.18	0.97
Minimum latency ms	-50.00%	0.01	0.02
Maximum latency ms	-7.65%	22.1	23.93
Stdev latency ms	25.93%	2.04	1.62
Transfer Mb	7.30%	687.05	640.3
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-06-10T12:34:24Z

A new generated diff is ready to view.

A new doc preview is ready to view.

github-actions · 2022-06-10T12:59:43Z

A new generated diff is ready to view.

A new doc preview is ready to view.

github-actions · 2022-06-12T15:41:41Z

A new generated diff is ready to view.

A new doc preview is ready to view.

github-actions · 2022-06-13T09:41:18Z

A new generated diff is ready to view.

A new doc preview is ready to view.

david-perez

2nd pass. Much better regarding code duplication. We should invest into reducing the remaining bits of duplication via decorators, CodegenTarget, or more inheritance in future PRs.

...kotlin/software/amazon/smithy/rust/codegen/server/python/smithy/PythonCodegenServerPlugin.kt

david-perez · 2022-06-13T14:28:04Z

...kotlin/software/amazon/smithy/rust/codegen/server/python/smithy/PythonCodegenServerPlugin.kt

@@ -61,6 +60,9 @@ class RustCodegenServerPlugin : SmithyBuildPlugin {
            symbolVisitorConfig: SymbolVisitorConfig = DefaultConfig
        ) =
            SymbolVisitor(model, serviceShape = serviceShape, config = symbolVisitorConfig)
+                // Rename a set of symbols that do not implement `PyClass` and have been wrapped in
+                // `aws_smithy_http_server_python::types`.
+                .let { PythonServerSymbolProvider(it) }


We'll also need a symbol provider that renames reserved Python keywords?

I don't think so. We are not generating any pure python code, everything is shared through the CFFI layer, hence I don't see anything we should really rename here.

david-perez · 2022-06-13T14:29:05Z

...otlin/software/amazon/smithy/rust/codegen/server/python/smithy/PythonServerCodegenVisitor.kt

+class PythonServerCodegenVisitor(context: PluginContext, private val codegenDecorator: RustCodegenDecorator) :
+    ServerCodegenVisitor(context, codegenDecorator) {
+
+    init {


We could have a function that given a symbol provider, sets up the rest of the stuff.

...are/amazon/smithy/rust/codegen/server/python/smithy/generators/PythonApplicationGenerator.kt

david-perez · 2022-06-13T14:34:47Z

...are/amazon/smithy/rust/codegen/server/python/smithy/generators/PythonServerAttributeUtils.kt

+import software.amazon.smithy.rust.codegen.server.python.smithy.PythonServerCargoDependency
+
+/**
+ * This module contains utilities to render PyO3 attributes.


Are these extension methods really necessary? It think you get the same syntax sugar by using the Attribute class from RustTypes and calling render.

You are probably right, but I think having these methods in their own separate file is actually nice and I think this file will grow over time with new helpers.. If don't have strong opinions on this, if you feel we should get rid of them I can try to just use the Attribute class.

I have tried to use the Attribute class to render pyo3 attributes, but it messes up the code by removing the // of any doc block that is rendered just after it and I don't really understand why.

I will add an issue to this block of code to understand what is going on and try to move to the Attribute class at a later stage.

It looks like the issue comes from https://github.com/awslabs/smithy-rs/blob/15e1b3425f56ff2606121e99dff11de38424c744/codegen/src/main/kotlin/software/amazon/smithy/rust/codegen/rustlang/RustTypes.kt#L377

This line uses writer.raw(). If I change the line from writer.raw("#$bang[$annotation]") to writer.rust("##$bang[$annotation]" it start rendering as I would expect.

I am not sure of the implications of making this change..

...amazon/smithy/rust/codegen/server/python/smithy/generators/PythonServerStructureGenerator.kt

david-perez · 2022-06-13T14:45:00Z

...amazon/smithy/rust/codegen/server/python/smithy/generators/PythonServerStructureGenerator.kt

+            "SmithyPython" to PythonServerCargoDependency.SmithyHttpServerPython(codegenContext.runtimeConfig).asType()
+        )
+
+    override fun renderStructure() {


I think tacking on attributes is best accomplished through decorators. If the structures we're generating are fundamentally different / additional, then inheritance, CodegenTarget or splitting are better approaches.

...amazon/smithy/rust/codegen/server/python/smithy/generators/PythonServerStructureGenerator.kt

david-perez · 2022-06-13T14:48:16Z

rust-runtime/aws-smithy-http-server-python/src/error.rs

+    /// Custom error.
+    #[error("{0}")]
+    Custom(String),
+    /// Implement `From<pyo3::PyErr>`.


Are the Rustdoc comments accurate? Starting with "Implement" I mean.

Regarding thiserror, why are we not using it in other places? @rcoh

Fixed Rustdoc comments.

...oftware/amazon/smithy/rust/codegen/server/smithy/customizations/AdditionalErrorsDecorator.kt

github-actions · 2022-06-13T15:01:03Z

A new generated diff is ready to view.

A new doc preview is ready to view.

Co-authored-by: david-perez <d@vidp.dev>

github-actions · 2022-06-14T12:07:36Z

A new generated diff is ready to view.

A new doc preview is ready to view.

github-actions · 2022-06-14T13:12:22Z

A new generated diff is ready to view.

A new doc preview is ready to view.

github-actions · 2022-06-14T15:01:00Z

A new generated diff is ready to view.

A new doc preview is ready to view.

jdisanti

LGTM!

github-actions · 2022-06-14T15:52:54Z

A new generated diff is ready to view.

A new doc preview is ready to view.

crisidev added 8 commits May 21, 2022 22:15

Partial structure generation

4367c00

Add Python server decorator

8cca266

Wrap Blob type with PyClass

49a7656

Add documentation to PythonServerCodegenDecorator

cc46039

Generate structures, wrap blob types and implement SymbolProvider.

5e68c85

Generate errors and implement error conversion. Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

Codegenerate python operation implementation

8afe768

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

Add autogeneration of all operation handlers and routers

95d128f

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

Remove leftovers

95d9db2

crisidev mentioned this pull request May 23, 2022

[WIP] Generate server implementation for pure Python business logic #1367

Open

34 tasks

Update documentation

bc6afd7

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

crisidev added 2 commits May 23, 2022 16:57

Refactor server to be fully code generated

e6d2d53

Merge branch 'main' into oxipy-struct-codegen

52881c1

crisidev added enhancement New feature or request server Rust server SDK labels May 23, 2022

crisidev added 2 commits May 23, 2022 18:40

Fix documentation

1585aa7

Merge branch 'main' into oxipy-struct-codegen

7cf4615

Merge branch 'main' into oxipy-struct-codegen

8b36d85

Add enum generation

3082c83

crisidev commented May 24, 2022

View reviewed changes

codegen-server-test/python/build.gradle.kts Show resolved Hide resolved

Merge branch 'main' into oxipy-struct-codegen

8e90b1a

Improve symbol matching when replacing types.

45ed6b6

Add decorator to add `use aws_smithy_http_server_python::types` to lib.rs.

Fix string representation in Python

28c4a06

crisidev requested a review from david-perez June 10, 2022 12:34

Remove leftover

893c3cd

Merge branch 'main' into oxipy-struct-codegen

4cdfd67

Merge branch 'main' into oxipy-struct-codegen

d8a1a40

crisidev added 2 commits June 13, 2022 15:48

Remove tower tracing layer to prevent logging of sensitive stuff

ef8b592

Merge branch 'main' into oxipy-struct-codegen

e4d2b4e

david-perez reviewed Jun 13, 2022

View reviewed changes

david-perez approved these changes Jun 13, 2022

View reviewed changes

david-perez reviewed Jun 13, 2022

View reviewed changes

...oftware/amazon/smithy/rust/codegen/server/smithy/customizations/AdditionalErrorsDecorator.kt Show resolved Hide resolved

crisidev and others added 3 commits June 13, 2022 17:41

Apply suggestions from code review

b314f1c

Co-authored-by: david-perez <d@vidp.dev>

Merge branch 'main' into oxipy-struct-codegen

276519b

Use RuntimeType.From for impl From

15e1b34

Fix documentation

ec2ad28

crisidev added 2 commits June 14, 2022 15:40

Refactor StructureGenerator to promote code reuse for Python

8fe764e

Merge branch 'main' into oxipy-struct-codegen

4240344

jdisanti approved these changes Jun 14, 2022

View reviewed changes

Merge branch 'main' into oxipy-struct-codegen

50ba6cb

crisidev enabled auto-merge (squash) June 14, 2022 15:45

crisidev merged commit 8e84ee2 into main Jun 14, 2022

crisidev deleted the oxipy-struct-codegen branch June 14, 2022 16:16

crisidev mentioned this pull request Jun 28, 2022

SERVER_CHANGELOG.md has been added for v0.44.0 #1500

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Codegenerate StructureShape, BlobShape, application, server and Python runtime #1403

Codegenerate StructureShape, BlobShape, application, server and Python runtime #1403

crisidev commented May 23, 2022

github-actions bot commented May 23, 2022

github-actions bot commented May 23, 2022

github-actions bot commented May 23, 2022

github-actions bot commented May 23, 2022

github-actions bot commented May 23, 2022

github-actions bot commented May 24, 2022

github-actions bot commented May 24, 2022

github-actions bot commented May 24, 2022

github-actions bot commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

github-actions bot commented Jun 12, 2022

github-actions bot commented Jun 13, 2022

david-perez left a comment

david-perez Jun 13, 2022

crisidev Jun 13, 2022

david-perez Jun 13, 2022

david-perez Jun 13, 2022

crisidev Jun 13, 2022

crisidev Jun 14, 2022

crisidev Jun 14, 2022 •

edited

Loading

crisidev Jun 14, 2022

david-perez Jun 13, 2022

david-perez Jun 13, 2022

david-perez Jun 13, 2022

crisidev Jun 14, 2022

github-actions bot commented Jun 13, 2022

github-actions bot commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

jdisanti left a comment

github-actions bot commented Jun 14, 2022

Codegenerate StructureShape, BlobShape, application, server and Python runtime #1403

Codegenerate StructureShape, BlobShape, application, server and Python runtime #1403

Conversation

crisidev commented May 23, 2022

github-actions bot commented May 23, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 23, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 23, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 23, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 23, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 24, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 24, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented May 24, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

github-actions bot commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

github-actions bot commented Jun 12, 2022

github-actions bot commented Jun 13, 2022

david-perez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crisidev Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 13, 2022

github-actions bot commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

jdisanti left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 14, 2022

crisidev Jun 14, 2022 •

edited

Loading