Skip to content

service protocol build control #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 5, 2021
Merged

service protocol build control #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b9fea61
Update readme for new build configuration options
kggilmer May 5, 2021
ea55425
Allow specification of inclusion and exclusion for services. Add abil…
kggilmer May 5, 2021
a36affc
Cleanup
kggilmer May 5, 2021
989dee4
Update docs from PR feedback
kggilmer May 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 38 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ To see list of all projects run `./gradlew projects`
See the local.properties definition above to specify this in a config file.

```sh
./gradlew -Paws.services=lambda :codegen:sdk:bootstrap
./gradlew -Paws.services=+lambda :codegen:sdk:bootstrap
Copy link
Contributor

@aajtodd aajtodd May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question

should we retain + as optional? It would maintain the current behavior and make e.g. generating a handful of services less ceremonious aws.services=+lambda,+s3,+iam

Copy link
Contributor Author

@kggilmer kggilmer May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking about that too. I went back and forth but slightly favor requiring requiring the plus for these reasons:

  1. generally speaking less configuration variance is easier to maintain over time (very weak admittedly in this case)
  2. the + self-documents the feature that exclusions are also available.
  3. it may be less confusing when reasoning about the rules if the ex/in-lusion is explicit.

IMHO these are a bit stronger than the convenience argument

```

##### Testing Locally
Expand All @@ -65,20 +65,51 @@ This will output HTML formatted documentation to `build/dokka/htmlMultiModule`

NOTE: You currently need an HTTP server to view the documentation in browser locally. You can either use the builtin server in Intellij or use your favorite local server (e.g. `python3 -m http.server`). See [Kotlin/dokka#1795](https://github.com/Kotlin/dokka/issues/1795)

### Build properties
### Build Properties
Copy link
Contributor

@aajtodd aajtodd May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

The intent of this section was to document the available build properties. The new section you added with the examples is fine and can remain as is but I still think we ought to document the available properties in one place.

Copy link
Contributor Author

@kggilmer kggilmer May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I added a table that describes all available properties directly in this section.


You can define a `local.properties` config file at the root of the project to modify build behavior.

An example config with the various properties is below:
|Property|Description|
|---|---|
|`compositeProjects`|Specify paths to repos the SDK depends upon such as `smithy-kotlin`|
|`aws.services`|Specify inclusions (+ prefix) and exclusions (- prefix) of service names to generate|
|`aws.protocols`|Specify inclusions (+ prefix) and exclusions (- prefix) of AWS protocols to generate|

```
#### Composite Projects

Dependencies of the SDK can be added as composite build such that multiple repos may appear as one
holistic source project in the IDE.

```ini
# comma separated list of paths to `includeBuild()`
# This is useful for local development of smithy-kotlin in particular
compositeProjects=../smithy-kotlin
```

#### Generating Specific Services Based on Name or Protocol

# comma separated list of services to generate from codegen/sdk/aws-models. When not specified all services are generated
# service names match the filenames in the models directory `service.VERSION.json`
aws.services=lambda
A comma separated list of services to include or exclude for generation from codegen/sdk/aws-models may
be specified with the `aws.services` property. A list of protocols of services to generate may be specified
with the `aws.protocols` property.

Included services require a '+' character prefix and excluded services require a '-' character.
If any items are specified for inclusion, only specified included members will be generated. If no items
are specified for inclusion, all members not excluded will be generated.
When unspecified all services found in the directory specified by the `modelsDir` property are generated.
Service names match the filenames in the models directory `service.VERSION.json`.

Some example entries for `local.properties`:
```ini
# Generate only AWS Lambda:
aws.services=+lambda
```

```ini
# Generate all services but AWS location and AWS DynamoDB:
aws.services=-location,-dynamodb
```

```ini
# Generate all services except those using the restJson1 protocol:
aws.protocols=-restJson1
```
67 changes: 61 additions & 6 deletions codegen/sdk/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,21 +110,36 @@ val sdkPackageNamePrefix = "aws.sdk.kotlin.services."
// Returns an AwsService model for every JSON file found in in directory defined by property `modelsDirProp`
fun discoverServices(): List<AwsService> {
val modelsDir: String by project
val serviceIncludeList = getProperty("aws.services")?.split(",")?.map { it.trim() }
val serviceMembership = parseMembership(getProperty("aws.services"))
val protocolMembership = parseMembership(getProperty("aws.protocols"))

return fileTree(project.file(modelsDir))
.filter {
val includeModel = serviceIncludeList?.contains(it.name.split(".").first()) ?: true
if (!includeModel) {
logger.info("skipping ${it.absolutePath}, name not included in aws.services")
.filter { file ->
val svcName = file.name.split(".").first()
val include = serviceMembership.isMember(svcName)

if (!include) {
logger.info("skipping ${file.absolutePath}, $svcName not a member of $serviceMembership")
}
includeModel
include
}
.map { file ->
val model = Model.assembler().addImport(file.absolutePath).assemble().result.get()
val services: List<ServiceShape> = model.shapes(ServiceShape::class.java).sorted().toList()
require(services.size == 1) { "Expected one service per aws model, but found ${services.size} in ${file.absolutePath}: ${services.map { it.id }}" }
val service = services.first()
file to service
}
.filter { (file, service) ->
val protocol = service.protocol()
val include = protocolMembership.isMember(protocol)

if (!include) {
logger.info("skipping ${file.absolutePath}, $protocol not a member of $protocolMembership")
}
include
}
.map { (file, service) ->
val serviceApi = service.getTrait(software.amazon.smithy.aws.traits.ServiceTrait::class.java).orNull()
?: error { "Expected aws.api#service trait attached to model ${file.absolutePath}" }
val (name, version, _) = file.name.split(".")
Expand All @@ -147,6 +162,46 @@ fun discoverServices(): List<AwsService> {
}
}

// Returns the trait name of the protocol of the service
fun ServiceShape.protocol(): String =
listOf(
"aws.protocols#awsJson1_0",
"aws.protocols#awsJson1_1",
"aws.protocols#awsQuery",
"aws.protocols#ec2Query",
"aws.protocols#ec2QueryName",
"aws.protocols#restJson1",
"aws.protocols#restXml"
).first { protocol -> findTrait(protocol).isPresent }.split("#")[1]

// Class and functions for service and protocol membership for SDK generation
data class Membership(val inclusions: Set<String> = emptySet(), val exclusions: Set<String> = emptySet())
fun Membership.isMember(member: String): Boolean = when {
exclusions.contains(member) -> false
inclusions.contains(member) -> true
inclusions.isEmpty() -> true
else -> false
}
fun parseMembership(rawList: String?): Membership {
if (rawList == null) return Membership()

val inclusions = mutableSetOf<String>()
val exclusions = mutableSetOf<String>()

rawList.split(",").map { it.trim() }.forEach { item ->
when {
item.startsWith('-') -> exclusions.add(item.substring(1))
item.startsWith('+') -> inclusions.add(item.substring(1))
else -> error("Must specify inclusion (+) or exclusion (-) prefix character to $item.")
}
}

val conflictingMembers = inclusions.intersect(exclusions)
require(conflictingMembers.isEmpty()) { "$conflictingMembers specified both for inclusion and exclusion in $rawList" }

return Membership(inclusions, exclusions)
}

fun <T> java.util.Optional<T>.orNull(): T? = this.orElse(null)

/**
Expand Down