ci: build all AWS services for PRs to main #190
Conversation
| import sys | ||
| import os | ||
|
|
||
| not_supported_yet = ["ec2", "location", "elasticbeanstalk", "marketplacecommerceanalytics"] |
There was a problem hiding this comment.
Correctness: Elastic Beanstalk should be supported now.
There was a problem hiding this comment.
marketplacecommerceanalytics can join the party as well.
| find codegen/sdk/aws-models -name '*.json' | sort | awk "NR % $JOBS == $JOB - 1" > services.txt | ||
| echo "Building the following services" | ||
| cat services.txt | ||
| ./gradlew --no-daemon :codegen:sdk:bootstrap -Paws.services=$(cat services.txt | .github/scripts/generate-includes.py) |
There was a problem hiding this comment.
Question: Why is the generate-includes.py script necessary if we can specify which protocols/services not to build? Won't this be equivalent and simpler?
./gradlews -Paws.services=-location,-marketplacecommerceanalytics -Paws.protocols=-ec2Query :codegen:sdk:bootstrapThere was a problem hiding this comment.
This is executed in batches. We only want to build n services per runner (where n = 1/NUM_JOBS). So the include list enforces that only those services even exist in the build. Honestly, this can all evolve I just wanted to get something that runs in a reasonable amount of time in the short term.
| ```sh | ||
| ./gradlew :codegen:sdk:bootstrap | ||
| ./gradlew --no-daemon :codegen:sdk:bootstrap | ||
| ``` |
There was a problem hiding this comment.
Question: Why are we recommending --no-daemon now?
There was a problem hiding this comment.
It should help prevent having to call ./gradlew --stop before executing.
| @@ -1,10 +1,13 @@ | |||
| kotlin.code.style=official | |||
| kkotlin.incremental.js=true | |||
There was a problem hiding this comment.
Correctness: kkotlin → kotlin
kggilmer
left a comment
kggilmer
left a comment
There was a problem hiding this comment.
Nice work on raising the bar for build correctness along w/ this CI addition.
| with: | ||
| name: m2-${{ env.name }}-${{ github.sha }} | ||
| path: ~/.m2 | ||
| retention-days: 2 |
There was a problem hiding this comment.
question
why 2 days here? I assume this repo caching is particular to a given PR. Would this be for day boundries?
There was a problem hiding this comment.
We don't really need the artifact past building the AWS services in parallel. 2 days seemed reasonable to me to debug and diagnose any build failures.
| - name: Generate SDKs | ||
| run: | | ||
| ls -lsa ~/.m2/repository | ||
| find codegen/sdk/aws-models -name '*.json' | sort | awk "NR % $JOBS == $JOB - 1" > services.txt |
There was a problem hiding this comment.
question
I take it this is what breaks the work up for each worker?
There was a problem hiding this comment.
Yes, it splits the list of services into batches modulo the number of "jobs".
| dependsOn(tasks["generateSmithyBuild"]) | ||
| inputs.file(projectDir.resolve("smithy-build.json")) | ||
| // ensure smithy-aws-kotlin-codegen is up to date | ||
| inputs.files(configurations.compileClasspath) |
There was a problem hiding this comment.
comment
very nice! I was trying to get these gradle configurations to work at one point but couldn't make it work. Well done!
There was a problem hiding this comment.
This seems to solve it but let me know if you see issues still. For me it seemed to rebuild appropriately now. However, I still have issues where I have to call ./gradlew --stop but at least everything is up-to date. Running codegen with --no-daemon seems to help some FWIW
|
|
||
| // force rebuild every time while developing | ||
| tasks["buildSdk"].outputs.upToDateWhen { false } | ||
| tasks["generateSdk"].outputs.upToDateWhen { false } |
3 participants
Issue #
Description of changes
smithy-kotlinCI. This isn't insurmountable but it is quite a bit more work. Getting the build times reasonable required implementing the build as a workflow which makes it hard to re-use.dependenciesdeclared are up to date. Changes to codegen should now automatically be rebuilt when you goto (re)generate code. There is still an issue with smithy itself and caching behavior where it requires a./gradlew --stopcall but at least now you can be sure codegen is up to date when you generate.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.