Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS::CodeBuild::Project - Support Batch Builds (From Trigger) #621

Closed
wulfmann opened this issue Sep 3, 2020 · 17 comments
Closed

AWS::CodeBuild::Project - Support Batch Builds (From Trigger) #621

wulfmann opened this issue Sep 3, 2020 · 17 comments
Labels
Coverage
Projects
coverage-roadmap

Comments

@wulfmann
Copy link

wulfmann commented Sep 3, 2020

1. AWS::CodeBuild::Project - Support Batch Builds

When creating a CodeBuild project there is no way to specify that the webhook trigger should run the project as a 'Batch Build'

2. Scope of request

a) new option for an existing attribute is desired

There is already a webhook option in the project triggers section of the provider, but it is a boolean value. It would seem that support for a new 'BuildType' option should be added.

3. Expected behavior

Allow me to specify a trigger type of STANDARD or BATCH

4. Suggest specific test cases

Samples:

Create a CodeBuild Project with a trigger type of STANDARD

Create a CodeBuild Project with a trigger type of BATCH

5. Helpful Links to speed up research and evaluation

Trigger batch build with cli

6. Category (required)

  1. Developer Tools (CodeBuild)
@wulfmann
Copy link
Author

wulfmann commented Sep 3, 2020

I can include screenshots from the console if that is helpful.

@wulfmann wulfmann changed the title AWS::CodeBuild::Project - Support Batch Builds AWS::CodeBuild::Project - Support Batch Builds (From Trigger) Sep 3, 2020
@tjenkinson
Copy link

It's not supported at the moment unfortunately. If you're using the CDK it's possible to create a custom resource using AwsCustomResource which worked for us. Using the CodeBuild service and createWebhook action

@wulfmann
Copy link
Author

Thanks for the tip @tjenkinson ! That seems like it will be adequate until this ticket is worked.

@subinataws
Copy link

@wulfmann - Thanks for reporting this issue. Team is working on a fix and will address it in the next week or so.

@tjenkinson
Copy link

just opened #633 which is support for triggering batch builds from codepipeline

@tjenkinson
Copy link

@subinataws any update on this?

@johnhanks1
Copy link

Hello @tjenkinson

We have enabled this for projects. This can be used with CloudFormation with the BuildType option, the valid inputs are BUILD or BUILD_BATCH example below

Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Ref ProjectName
      ...
      Source:
        Type: GITHUB
        Location: !Ref SourceLocation
        InsecureSsl: true
        GitCloneDepth: 1
        ReportBuildStatus: false
      Triggers:
        Webhook: true
        BuildType: BUILD_BATCH

We are working on updating or official documentation. Sorry for the confusion.

@tjenkinson tjenkinson mentioned this issue Nov 24, 2020
Closed
2 tasks
@tjenkinson
Copy link

That's great thanks. Next step is getting it into the CDK :)

@tjenkinson tjenkinson mentioned this issue Nov 27, 2020
Merged
@tjenkinson
Copy link

@johnhanks1 does this setting apply for any trigger or just webhooks?

Question from @skinny85 on the cdk PR (aws/aws-cdk#11743 (comment))

So this setting works only for builds started through the webhook? Not through, for instance, CloudWatch Events?

@tjenkinson
Copy link

Found it in the docs now but it says

Not currently supported by AWS CloudFormation.

@tjenkinson
Copy link

@johnhanks1 would be great to get some clarity on this because it's blocking the CDK PR. Thanks!

mergify bot pushed a commit to aws/aws-cdk that referenced this issue Dec 24, 2020
@tjenkinson
feat(codebuild): add startBatchBuild option (#11743)
d9353b7 
This adds a `startBatchBuild` option to the code build source, to trigger a batch build. The cloudformation property isn't in the official docs yet but is mentioned [here](aws-cloudformation/cloudformation-coverage-roadmap#621 (comment)).

Closes #11663
flochaz pushed a commit to flochaz/aws-cdk that referenced this issue Jan 5, 2021
@tjenkinson @flochaz
feat(codebuild): add startBatchBuild option (aws#11743)
f2a9950 
This adds a `startBatchBuild` option to the code build source, to trigger a batch build. The cloudformation property isn't in the official docs yet but is mentioned [here](aws-cloudformation/cloudformation-coverage-roadmap#621 (comment)).

Closes aws#11663
@paulsbnet
Copy link

Hello @tjenkinson

We have enabled this for projects. This can be used with CloudFormation with the BuildType option, the valid inputs are BUILD or BUILD_BATCH example below

Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Ref ProjectName
      ...
      Source:
        Type: GITHUB
        Location: !Ref SourceLocation
        InsecureSsl: true
        GitCloneDepth: 1
        ReportBuildStatus: false
      Triggers:
        Webhook: true
        BuildType: BUILD_BATCH

We are working on updating or official documentation. Sorry for the confusion.

Howdy!

I attempted to update the codebuild project I am working on that requires batch builds in order to build properly and it seems that its still not executing when I run the buildspec.yml that I declare the batch in.
I used the awscli to pull the build project information and saw that the cloudformation update did work and the buildType was set to BUILD_BATCH.
Is this feature fully tested and working at this point or is there more to be done? It seems as though I have all the required parts to get this feature working.

Thanks for any assistance that can be provided.

@tjenkinson
Copy link

Does the config in the console show it's set to the batch mode?

@paulsbnet
Copy link

paulsbnet commented Jan 6, 2021 •
edited

Does the config in the console show it's set to the batch mode?

Managed to get it working.
Needed to add these to the cloudformation in order for it to work:

      BuildBatchConfig:
        CombineArtifacts: true
        ServiceRole: !GetAtt Role.Arn

Thank you for your time though!

@tjenkinson tjenkinson mentioned this issue Jan 15, 2021
Merged
mergify bot pushed a commit to aws/aws-cdk that referenced this issue Jan 26, 2021
@tjenkinson
feat(aws-codebuild): add enableBatchBuilds() to Project (#12531)
0568390 
In order for a CodeBuild to run in batch mode, a batch service role is needed, as described [here in the docs](https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build.html).

>Batch builds introduce a new security role in the batch configuration. This new role is required as CodeBuild must be able to call the StartBuild, StopBuild, and RetryBuild actions on your behalf to run builds as part of a batch. Customers should use a new role, and not the same role they use in their build...

At first I thought lets add this by default, but then I realised when `BatchConfiguration` is set to something, in the aws console the default 'start build' button behaviour changes to start a batch build by default instead :/

So now this adds a new `supportBatchBuildType` option, which when `true` adds minimum amount of `BatchConfiguration` needed for batch builds to run.

I also updated the doc blocks for the webhook option and CodePipeline action option, because users of those also need to set this option. It would be nice to auto-enable this if a webhook or CodeBuild action is configured, but that sounds pretty complicated.

I'm not sure why anyone would need to customise this role, given it appears to only be used internally to do those 3 things, so this PR does not make it configurable. My thinking is that this could be added later if needed, but this PR just gets batch builds working.

In the future if people want control of the other `BatchConfiguration` options I was thinking these could be added and would require `supportBatchBuildType` to be `true`.

related: aws-cloudformation/cloudformation-coverage-roadmap#621
@tscully49
Copy link

Does this work with the local codebuild agent docker image? I can't seem to get it to kick off batch builds thus far.

@WaelA WaelA added the Coverage label Aug 3, 2021
@cfn-github-issues-bot cfn-github-issues-bot added this to Researching in coverage-roadmap Aug 19, 2021
@cfn-github-issues-bot cfn-github-issues-bot moved this from Researching to We're working on it in coverage-roadmap Sep 20, 2021
@cfn-github-issues-bot cfn-github-issues-bot moved this from We're working on it to Coming Soon in coverage-roadmap Sep 20, 2021
@jigar-lab
Copy link

This is now supported from CFN and CDK both.

@jigar-lab
Copy link

jigar-lab commented Sep 20, 2021 •
edited

Does this work with the local codebuild agent docker image? I can't seem to get it to kick off batch builds thus far.

Thanks for reporting this. We have added a backlog item to assess the feasibility and add batch build support in local codebuild agent.

@cfn-github-issues-bot cfn-github-issues-bot moved this from Coming Soon to Shipped in coverage-roadmap Feb 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Coverage
Projects
coverage-roadmap
  
Shipped
Milestone
No milestone
Development

No branches or pull requests
9 participants
@johnhanks1 @tjenkinson @wulfmann @tscully49 @WaelA @paulsbnet @subinataws @jigar-lab @cfn-github-issues-bot