Add sphinx doc generation #418

SamRemis · 2025-03-10T17:57:06Z

This PR adds generation of sphinx docs. There are a number of changes that are necessary to accomplish this:

Converting the HTML provided by service teams to RST for sphinx (AwsDocConverter.java)
Updating the linewrapping to respect RST formatting (PythonWriter.java)
Add the core sphinx files to be generated during the initial repo setup (SetupGenerator.java)
Add events during client generation for when we are generating a symbol that needs to be documented (many files- most of the *Generator.java files as well as the *Section.java files)
Add more necessary boiler plate code- requirements, references to the new interceptors, etc.

This is still in draft mode pending the following action items:

Add tests

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

.../core/src/main/java/software/amazon/smithy/python/codegen/generators/StructureGenerator.java

codegen/aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsDocConverter.java

codegen/core/src/main/java/software/amazon/smithy/python/codegen/writer/PythonWriter.java

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/UnionGenerator.java

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/SetupGenerator.java

codegen/core/src/main/java/software/amazon/smithy/python/codegen/writer/PythonWriter.java

nateprewitt · 2025-03-17T20:35:49Z

codegen/core/src/main/java/software/amazon/smithy/python/codegen/writer/PythonWriter.java

+            // If no space found, force wrap at maxLineLength
+            wrapAt = maxLineLength;


How does this work for things like links and code that may stretch beyond the limit. Is this going to create rendering issues?

It does create some; with the current docs there is only one link I can find with this issue.

What is the ideal behavior here? Should we allow these docs to exceed the max line length?

If it's a link, probably. That only affects rendering in the code. There's a question of if we even want the links in the docstrings, but I don't think we need to solve that now.

pyproject.toml

Draft commit - whitespace changes still need to be addressed

JordonPhillips · 2025-03-19T11:20:00Z

codegen/aws/core/build.gradle.kts

 dependencies {
    implementation(project(":core"))
    implementation(libs.smithy.aws.traits)
+    implementation("org.jsoup:jsoup:1.19.1")


The version should go in here. Docs on how that works are here.

JordonPhillips · 2025-03-19T11:32:31Z

codegen/aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsDocConverter.java

+    }
+
+    private static class RstNodeVisitor implements NodeVisitor {
+        private final StringBuilder sb = new StringBuilder();


It'll probably be better to use a SimpleCodeWriter for this. There's a few major benefits. For starters, it will make indentation management easier. Right now any list values that happen to have a newline will break the indentation. Same for anything in any sort of block.

A secondary benefit is that you can then create a named section that can be pushed in the opening tag and popped in the closing tag. That'll let you intercept those sections later down the line if you need to do customizations. It'll also let you support ordered lists since you can put context into the writer to say which list element you're on. Since context is scoped to sections, you won't have to worry about overriding some parent.

Given time constraints we have right now, this can be a fast follow.

JordonPhillips · 2025-03-19T11:42:50Z

codegen/aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsDocConverter.java

+                DocumentationTrait docTrait = shape.getTrait(DocumentationTrait.class).get();
+                String html = docTrait.getValue();
+                String rst = convertHtmlToRst(html);
+                DocumentationTrait newDocTrait = new DocumentationTrait(rst);


I'd rather run this in PythonWrtier where we write documentation. The format of the documentation trait is commonmark, so putting RST in there isn't really correct.

Speaking of commonmark, you need to account for that. The easiest way is to just render the whole thing to HTML like this. commonmark can contain HTML (and most of the docs we get from AWS are just HTML) but there's more to it.

JordonPhillips · 2025-03-19T11:45:59Z

...aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsRstDocFileGenerator.java

+        return String.format("%s%n%s%n%n", title, "=".repeat(title.length()));
+    }
+
+    private static final class OperationGenerationInterceptor


What's the advantage of doing this over just autoclassing the client itself?

JordonPhillips · 2025-03-19T11:47:04Z

codegen/aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsDocConverter.java

This doesn't need to be AWS specific, nor does the rst doc file generator

JordonPhillips · 2025-03-19T11:52:03Z

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/SetupGenerator.java

+    /**
+     * Write the files required for sphinx doc generation
+     */
+    private static void writeDocsSkeleton(


We don't need to do this now, but lets add a todo to make this configurable. We should be able to turn off generating all the sphinx setup files and select a theme.

JordonPhillips · 2025-03-19T11:54:17Z

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/SetupGenerator.java

+                    import sys
+                    sys.path.insert(0, os.path.abspath('..'))
+
+                    project = 'AWS SDK for Python'


This should reference the service title, falling back to the shape name. You could preprocess the model in the AWS side to set the title trait to be based on the SDK ID if you like.

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/UnionGenerator.java

…en/generators/SetupGenerator.java Co-authored-by: Nate Prewitt <nate.prewitt@gmail.com>

JordonPhillips

Once this and that bug are fixed I'm approving

...re/src/main/java/software/amazon/smithy/python/codegen/writer/MarkdownToRstDocConverter.java

nateprewitt reviewed Mar 10, 2025

View reviewed changes

.../core/src/main/java/software/amazon/smithy/python/codegen/generators/StructureGenerator.java Outdated Show resolved Hide resolved

SamRemis force-pushed the convert-docs-to-rst branch from a0c24c0 to 0036e2e Compare March 17, 2025 16:58

SamRemis commented Mar 17, 2025

View reviewed changes

codegen/aws/core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsDocConverter.java Outdated Show resolved Hide resolved

SamRemis commented Mar 17, 2025

View reviewed changes

codegen/core/src/main/java/software/amazon/smithy/python/codegen/writer/PythonWriter.java Show resolved Hide resolved

codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/UnionGenerator.java Show resolved Hide resolved

nateprewitt reviewed Mar 17, 2025

View reviewed changes

SamRemis changed the title ~~Convert HTML to RST in doc models~~ Add sphinx doc generation Mar 17, 2025

SamRemis force-pushed the convert-docs-to-rst branch from 764264e to f1b9ced Compare March 18, 2025 15:08

SamRemis added 4 commits March 18, 2025 13:32

Initial commit

7d85d74

Draft commit - whitespace changes still need to be addressed

Add RST file generation, RST linewrapping, and whitespace fixes

5ad44ff

Run linter, minor fixes

3c637c6

Fix linewrapping, add tests

1a36552

SamRemis force-pushed the convert-docs-to-rst branch from f1b9ced to 1a36552 Compare March 18, 2025 18:19

remove global dependencies on sphinx and pydata

2eab1a8

SamRemis marked this pull request as ready for review March 18, 2025 19:02

SamRemis requested a review from a team as a code owner March 18, 2025 19:02

Fix linewrapping issue

eb0f27b

JordonPhillips requested changes Mar 19, 2025

View reviewed changes

SamRemis and others added 2 commits March 19, 2025 18:08

Updates based on PR feedback

cf9e9ec

Update codegen/core/src/main/java/software/amazon/smithy/python/codeg…

fb551e2

…en/generators/SetupGenerator.java Co-authored-by: Nate Prewitt <nate.prewitt@gmail.com>

JordonPhillips previously approved these changes Mar 19, 2025

View reviewed changes

Merge branch 'develop' into convert-docs-to-rst

7abe2ce

SamRemis dismissed JordonPhillips’s stale review via 7abe2ce March 20, 2025 13:29

Update libs.versions.toml

169c988

JordonPhillips previously approved these changes Mar 20, 2025

View reviewed changes

...re/src/main/java/software/amazon/smithy/python/codegen/writer/MarkdownToRstDocConverter.java Outdated Show resolved Hide resolved

variable fix

12f66d8

SamRemis dismissed JordonPhillips’s stale review via 12f66d8 March 20, 2025 15:56

Update MarkdownToRstDocConverterTest.java

25f2bd4

JordonPhillips approved these changes Mar 20, 2025

View reviewed changes

SamRemis merged commit ff78144 into smithy-lang:develop Mar 20, 2025
2 checks passed

arandito mentioned this pull request Oct 31, 2025

Replace Sphinx doc gen with MkDocs and Markdown #592

Open

		// If no space found, force wrap at maxLineLength
		wrapAt = maxLineLength;

Add sphinx doc generation #418

Add sphinx doc generation #418

Uh oh!

Conversation

SamRemis commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JordonPhillips left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SamRemis commented Mar 10, 2025 •

edited

Loading