SDK Modularization

[skmcgrail]

Background

At the general availability release of the AWS SDK for Go on November 19th 2015 the state of the Go language, compiler, and associated tools had just reached a significant milestone for the language. The language had only just reached version 1.5 in August that year, which brought with it major changes: a compiler and runtime fully written in Go, concurrent garbage collection, support for internal packages, experimental vendoring support,and other changes significant changes and improvements. The vendoring experiment was significant as it allowed Go package authors to to bundle an exact copy of their dependencies.

With the initial experimental introduction of modules in Go 1.11 and finalization in 1.13, library and application owners can now leverage a go.mod file for tracking dependent Go modules and respective minimum versions required to build the target module. The AWS SDK for Go uses a single module today with incremental versioning increases of its patch and minor versions. When users update their SDK version to a newer release this brings with it changes to all components and services vended by the SDK. This can create breaking changes to user applications. For example customers will experience failed code builds if utilizing the API interface types that are generated per service. Using versioned modules per service would allow customers to chose service API updates that result in the interface definition changing. Modularization of services would additionally allow for the SDK to release major version changes when changing the SDK's presentation of the service APIs.

Proposal

Vanity Import Path

Today customers use the SDK by importing the SDK into their application. For example to begin using the SDK with S3 users simply import the SDK using an import statement like import "github.com/aws/aws-sdk-go-v2/service/s3" and the Go tooling can retrieve the required code and associated dependencies for the user.

We propose the implementation a vanity path such as

import "sdk-go-v2.aws/sdk/service/s3"

This import statement would communicate upfront to SDK consumers that they are importing version 2 of the AWS SDK for Go. When Go encounters an unknown import path referring to an unknown hosting site the tooling will attempt to fetch the target path over HTTPS. The SDK import path would be configured to return an HTML <meta /> tag indicating to the Go tooling how the source artifacts can be retrieved.

<meta name="go-import" content="sdk-go-v2.aws/sdk git https://github.com/aws/aws-sdk-go-v2" />

This metadata tag indicates to the Go tooling that the given import path request can be retrieved using git at the repository URL https://github.com/aws/aws-sdk-go-v2.

Implementing a vanity path would provide flexibility in where and how code is retrieved, whether that is using a a single repository VCS, multi-repository VCS, or even what Go Module Proxy to talk to.

High Level Repository and Module Structure

Go supports a single-repository structure that contains multiple Go modules. For modularization of the V2 SDK we propose migrating much of the core supporting functionality for the AWS SDK into the root repository module. This would include much of the source code found in the aws package today. Additionally each service would be a Go module with a tagged version number that is updated independently. Moving to a module per service will allow for consumers of the SDK to not be forced to pick-up modeling changes for every service when performing incremental minor version upgrades.

sdk-go-v2.aws/sdk (Module)
|_ examples (Module)
|_ internal
|_ service
    |_ S3 (Module)
    |_ EC2 (Module)
|_ loadconfig (Module)
|_ credentials (Module)
|_ client

Versioning Modules

Go supports several methods for versioning modules. First Go determines the version of a given module by using the target repository VCS. In the case of Git, git tags are used to indicated semantic versions for creating module release points. For example a tag of v1.1.0 would indicate the root module sdk-go-v2.aws/sdk at that source version. A tag of service/S3/v1.2.0 would indicated version v1.2.0 for the S3 service sub-module.

To support newer SDK features that modify or break existing exported types or functions, we must create a new major version release for services that are affected. Go follows an import path style that is refereed to as "semantic import versioning" which requires that major versions at or greater then v2 include the versioning as the last component of the import path. So for example sdk-go-v2.aws/sdk/service/s3/v2 would become the import path for accessing the S3 client when significant changes have been made to exported types of functions. This major version change would occur by creating a v2 directory within the s3 module, and adding a
go.mod file to the v2 directory and updating the module path to become sdk-go-v2.aws/service/s3/v2 and lastly creating a git tag at service/s3/v2.Y.Z.

[samuelkarp]

Thanks for posting this here and asking for feedback! I have a few questions:

High Level Repository and Module Structure

What's the reasoning for moving each service to a separate module? This section covers a bit of the usage implication, but not the background as to why. From a usage perspective, this section also doesn't quite address what it looks like for an SDK consumer; each imported service is now going to appear in the go.mod file separately.

I know that other SDKs (including Java) are split up into separate packages or modules. For Java, there is an advantage to doing this: the size of the classpath affects the search time for classes that are loaded and thus the effective application startup time. But Go is statically-linked and unused packages are excluded from the binary at link time. Is there something here where a consumer of the SDK would benefit by having a modularized structure?

Vanity Import Path & Versioning Modules

1. If the vanity path is "sdk-go-v2.aws/sdk", the string "sdk" is repeated twice. Do we believe that having the second "sdk" is valuable there? Using the root ("sdk-go-v2.aws") doesn't preclude you from hosting content at that path; the meta tag can be present on a page with content as well (though I guess there is some concern over the size of the content, perhaps?).
2. This is the v2 of the SDK, but we don't currently host the existing SDK for Go with a vanity import path. Is the "v2" in the domain name valuable? My inclination is that the two SDKs will be distinguishable by import path, regardless of whether "v2" is actually present in that path. (And "v2" in the domain name can get confusing with Semantic Import Versioning.)
3. Once any given module under this vanity import reaches v2.0.0, a "v2" segment will need to be introduced to the import path. Given that this is "v2" of the SDK, does the GA versioning start at v2.0.0? If there is a "v2" both in the domain name and the path, that seems easy to confuse.

I think my preference would be for import paths to be structured more like this:

• sdk-go.aws/core
• sdk-go.aws/service/s3
• (when v2.0.0 happens) sdk-go.aws/v2/core (requires a single module at the top-level)

[smswz]

Just to throw another hat in the ring, but what about a more fluid URL with subdomains:
go.sdk.aws/core
go.sdk.aws/service/s3
go.sdk.aws/v2/service/s3

[jasdel]

Closing this issue as the v2 SDK v0.25.0 refresh updated the SDK to be modularized.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.