Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(glue): Add G.4X and G.8X worker types for AWS Glue #25637

Merged
merged 20 commits into from
May 24, 2023
Merged

feat(glue): Add G.4X and G.8X worker types for AWS Glue #25637

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
41721b1
Add G.4X and G.8X worker types for AWS Glue
racc May 18, 2023
36ab45d
Update unit test to add new worker types
racc May 18, 2023
709995a
Remove trailing space
racc May 18, 2023
33878a1
Update readme and integration tests
racc May 21, 2023
9f816fe
Update snapshot
racc May 21, 2023
8dbefa1
Add updated snapshot
racc May 21, 2023
d71d5b0
Fix readme
racc May 22, 2023
011672b
Add G.4X and G.8X worker types for AWS Glue
racc May 18, 2023
068d178
Update unit test to add new worker types
racc May 18, 2023
54e7da8
Remove trailing space
racc May 18, 2023
2439319
Update readme and integration tests
racc May 21, 2023
bc8eb4a
Update snapshot
racc May 21, 2023
c58a332
Add updated snapshot
racc May 21, 2023
8783650
Fix readme
racc May 22, 2023
925641a
Merge branch 'main' into main
racc May 22, 2023
e527967
Update following review
racc May 22, 2023
45fb79c
Merge branch 'main' of https://github.com/racc/aws-cdk
racc May 22, 2023
f87ee95
Add comma back following review
racc May 22, 2023
680df85
Merge branch 'main' into main
racc May 23, 2023
3ef5ed3
Merge branch 'main' into main
mergify[bot] May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packages/@aws-cdk/aws-glue-alpha/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ new glue.Job(this, 'ScalaSparkEtlJob', {
glueVersion: glue.GlueVersion.V4_0,
script: glue.Code.fromBucket(bucket, 'src/com/example/HelloWorld.scala'),
className: 'com.example.HelloWorld',
extraJars: [glue.Code.fromBucket(bucket, 'jars/HelloWorld.jar')],
extraJars: [glue.Code.fromBucket(bucket, 'jars/HelloWorld.jar')]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These kinds of commas are not typos, it makes it easier to extend without having to worry if it's the last line or higher up.

}),
workerType: glue.WorkerType.G_8X,
description: 'an example Scala ETL job',
});
```
Expand Down
10 changes: 10 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@ export class WorkerType {
*/
public static readonly G_2X = new WorkerType('G.2X');

/**
* Each worker maps to 4 DPU (16 vCPU, 64 GB of memory, 256 GB disk), and provides 1 executor per worker. We recommend this worker type for jobs whose workloads contain your most demanding transforms, aggregations, joins, and queries. This worker type is available only for AWS Glue version 3.0 or later jobs.
*/
public static readonly G_4X = new WorkerType('G.4X');

/**
* Each worker maps to 8 DPU (32 vCPU, 128 GB of memory, 512 GB disk), and provides 1 executor per worker. We recommend this worker type for jobs whose workloads contain your most demanding transforms, aggregations, joins, and queries. This worker type is available only for AWS Glue version 3.0 or later jobs.
*/
public static readonly G_8X = new WorkerType('G.8X');

/**
* Each worker maps to 0.25 DPU (2 vCPU, 4 GB of memory, 64 GB disk), and provides 1 executor per worker. Suitable for low volume streaming jobs.
*/
Expand Down
6 changes: 3 additions & 3 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job.js.snapshot/aws-glue-job.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@
"key": "value"
},
"Timeout": 5,
"WorkerType": "G.2X"
"WorkerType": "G.8X"
}
},
"EtlJob20SuccessMetricRule1759F889": {
Expand Down Expand Up @@ -547,7 +547,7 @@
"key": "value"
},
"Timeout": 5,
"WorkerType": "G.2X"
"WorkerType": "G.8X"
}
},
"EtlJob30SuccessMetricRuleF8870F8A": {
Expand Down Expand Up @@ -904,7 +904,7 @@
"key": "value"
},
"Timeout": 5,
"WorkerType": "G.2X"
"WorkerType": "G.8X"
}
},
"EtlJob40SuccessMetricRule00D3EF34": {
Expand Down
6 changes: 3 additions & 3 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job.js.snapshot/tree.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@
"key": "value"
},
"timeout": 5,
"workerType": "G.2X"
"workerType": "G.8X"
}
},
"constructInfo": {
Expand Down Expand Up @@ -808,7 +808,7 @@
"key": "value"
},
"timeout": 5,
"workerType": "G.2X"
"workerType": "G.8X"
}
},
"constructInfo": {
Expand Down Expand Up @@ -1324,7 +1324,7 @@
"key": "value"
},
"timeout": 5,
"workerType": "G.2X"
"workerType": "G.8X"
}
},
"constructInfo": {
Expand Down
2 changes: 1 addition & 1 deletion packages/@aws-cdk/aws-glue-alpha/test/integ.job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ const script = glue.Code.fromAsset(path.join(__dirname, 'job-script/hello_world.
glueVersion,
script,
}),
workerType: glue.WorkerType.G_2X,
workerType: glue.WorkerType.G_8X,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why make the tests 4x as expensive?

workerCount: 10,
maxConcurrentRuns: 2,
maxRetries: 2,
Expand Down
4 changes: 4 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/test/job.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ describe('WorkerType', () => {

test('.G_2X should set the name correctly', () => expect(glue.WorkerType.G_2X.name).toEqual('G.2X'));

test('.G_4X should set the name correctly', () => expect(glue.WorkerType.G_4X.name).toEqual('G.4X'));

test('.G_8X should set the name correctly', () => expect(glue.WorkerType.G_8X.name).toEqual('G.8X'));

test('.G_025X should set the name correctly', () => expect(glue.WorkerType.G_025X.name).toEqual('G.025X'));

test('.Z_2X should set the name correctly', () => expect(glue.WorkerType.Z_2X.name).toEqual('Z.2X'));
Expand Down