6593 improve usability of deploy new script by ca61688 · Pull Request #7606 · gchq/sleeper

ca61688 · 2026-07-06T13:08:46Z

Make sure you have checked all steps below.

Issue

My PR fully resolves the following issues. I've referenced an issue in the PR title, for example "Issue 1234 - My
Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
for your progress.
- Resolves Improve usability of deployNew.sh #6593

Tests

My PR adds the following tests based on our test strategy OR does not need testing for this extremely good reason:
- Manual testing including quick system test suite run

Documentation

In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
separate issue for that below.
If I have added new Java code, I have added Javadoc that explains it following our conventions and style.
If I have added or removed any dependencies from the project, I have updated the NOTICES file.

…s flag

…com:gchq/sleeper into 6593-improve-usability-of-deploy-new-script

patchwork01

I'm not sure what to do about it yet but I'm struggling to review this. We'd better talk about what we can do. I remember it was a pain to work with last time I tried to refactor it as well.

patchwork01 · 2026-07-15T13:01:35Z

-        deployPaused = builder.deployPaused;
+    public DeployNewInstance(InstanceDeployer deployInstance,
+            StoreFactory storeFactory,
+            SleeperInstanceConfiguration deployInstanceConfiguration,


I'm struggling to understand how this is passed around now, particularly as the path it comes from is passed as separate parameters alongside it.

This is easier to follow now, but I think it's still a bit confusing.

The InstanceProperties object is currently used as an intermediate data structure just to set the instance ID, VPC and subnets in the CdkCommand. I think that indirection is misleading, and I don't think there's a reason to mix that with the configuration from the files. Is it possible to build the CdkCommand directly from the Arguments object?

The InstanceProperties object is also used when we create the tables, but it's reloaded fresh from the deployed instance. I think it's confusing to reuse the same object for that, when it will contain very different data. We could use InstancePropertiesLoader instead of a reloadInstanceProperties.

That also means I don't think we need to load the whole configuration from the files, just the table properties. I think the only thing we're using the whole configuration for right now is to validate it, but the CDK will do that anyway.

There's also a tension with the original contract where DeployNewInstance is used elsewhere, because previously it was taking a DeployInstanceConfiguration and writing it to disk, but now it doesn't do that any more. We still need the callers to work. We can either just write it to disk if it was provided, or update the callers to write it to disk before they call this class.

Here's a link to the original thread: #7606 (comment)

patchwork01 · 2026-07-15T13:04:24Z

+    void setUp() throws IOException {
+        createTempDirectory(tempDir, null);
+        instancePropertiesFile = tempDir.resolve("instance.properties");
+        Files.writeString(instancePropertiesFile, instanceProperties.saveAsString());


This is generating an InstanceProperties object with InstancePropertiesTestHelper, and that simulates a pre-deployed instance. It's then writing it to a local file and treating it as a user's configuration file. I don't think it makes sense to treat one as the other, and I'm struggling to follow how these values are used.

Switched the instance properties being used to one without ID, VPC and Subnets set

There are a lot more properties that are set in createTestInstanceProperties that won't be set in a file provided by the user. It's still using a simulated pre-deployed instance, when what we want is something a user would write. We expect a user will set as few properties as possible. I think we can start from new InstanceProperties()?

Here's a link to the original thread: #7606 (comment)

…le or confgDir being set

…lProperties to create directories

…operties

rtjd6554 · 2026-07-23T13:53:26Z

+            Path directory, InstanceProperties instanceProperties,
+            Stream<TableProperties> tablePropertiesStream) throws IOException {
+        Files.createDirectories(directory);
+        FilesUtil.clearDirectory(directory);


Is this necessary, surely if its created it fresh then it should be empty?

createDirectories doesn't fail the directory already exists

Discussed together and happy with action

patchwork01 · 2026-07-31T08:27:36Z

+    void setUp() throws IOException {
+        createTempDirectory(tempDir, null);
+        instancePropertiesFile = tempDir.resolve("instance.properties");
+        Files.writeString(instancePropertiesFile, instanceProperties.saveAsString());


There are a lot more properties that are set in createTestInstanceProperties that won't be set in a file provided by the user. It's still using a simulated pre-deployed instance, when what we want is something a user would write. We expect a user will set as few properties as possible. I think we can start from new InstanceProperties()?

Here's a link to the original thread: #7606 (comment)

patchwork01 · 2026-07-31T08:47:42Z

        return builder().command(command).configurationDirectory(configurationDirectory).arguments(arguments).build();
    }

+    public CdkCommand withNetworkConfiguration(String instanceId, String vpcId, String subnets) {


This name seems slightly misleading, since the instance ID isn't part of the network configuration. I don't mind if you'd rather leave it as it is. Another option would be a toBuilder method.

patchwork01 · 2026-07-31T08:53:11Z

-        String subnetIds = args[3];
-        Path instancePropertiesFile = optionalArgument(args, 4).map(Path::of).orElse(null);
-        boolean deployPaused = "true".equalsIgnoreCase(optionalArgument(args, 5).orElse("false"));
+    public DeployNewInstance(Builder builder) {


By our coding conventions, the constructor should be the first method in the file, as it was before.

It looks like the constructor can be private.

patchwork01 · 2026-07-31T08:55:07Z

+        return config;
+    }
+
+    public static Arguments readArguments(CommandArguments arguments) {


I think it would be nice to keep this function right next to the usage constant, because you'll want to check it agrees with that about the names of the parameters. I think loadConfiguration can be below this.

patchwork01 · 2026-07-31T09:08:43Z

-        deployPaused = builder.deployPaused;
+    public DeployNewInstance(InstanceDeployer deployInstance,
+            StoreFactory storeFactory,
+            SleeperInstanceConfiguration deployInstanceConfiguration,


This is easier to follow now, but I think it's still a bit confusing.

The InstanceProperties object is currently used as an intermediate data structure just to set the instance ID, VPC and subnets in the CdkCommand. I think that indirection is misleading, and I don't think there's a reason to mix that with the configuration from the files. Is it possible to build the CdkCommand directly from the Arguments object?

The InstanceProperties object is also used when we create the tables, but it's reloaded fresh from the deployed instance. I think it's confusing to reuse the same object for that, when it will contain very different data. We could use InstancePropertiesLoader instead of a reloadInstanceProperties.

That also means I don't think we need to load the whole configuration from the files, just the table properties. I think the only thing we're using the whole configuration for right now is to validate it, but the CDK will do that anyway.

There's also a tension with the original contract where DeployNewInstance is used elsewhere, because previously it was taking a DeployInstanceConfiguration and writing it to disk, but now it doesn't do that any more. We still need the callers to work. We can either just write it to disk if it was provided, or update the callers to write it to disk before they call this class.

Here's a link to the original thread: #7606 (comment)

patchwork01 · 2026-07-31T09:36:31Z

+            config.getInstanceProperties().set(SUBNETS, subnets);
+        }
+
+        private DeployInstanceRequest buildExpectedCDKCommandWithPropertyFile(SleeperInstanceConfiguration config, boolean deployPaused) {


This is duplicating the implementation. Please make the asserted value explicit.

patchwork01 · 2026-07-31T09:36:45Z

+                    .build();
+        }
+
+        private DeployInstanceRequest buildExpectedCDKCommandWithConfigDir(SleeperInstanceConfiguration config, boolean deployPaused) {


This is duplicating the implementation. Please make the asserted value explicit.

patchwork01 · 2026-07-31T09:40:32Z

+            assertThat(deployRequests.size()).isEqualTo(1);
+            DeployInstanceRequest lastDeployRequest = deployRequests.get(0);
+            assertThat(lastDeployRequest).usingRecursiveComparison()
+                    .isEqualTo(buildExpectedCDKCommandWithPropertyFile(config, false));


This two assertions can be one, with assertThat(...).containsExactly.

I don't think usingRecursiveComparison is necessary? CdkCommand is a record.

This also applies to the other tests.

patchwork01 · 2026-07-31T09:46:22Z

+                    .storeFactory(StoreFactory.withAwsClients(s3Client, dynamoClient, accountName))
                    .deployInstanceConfiguration(config)
-                    .cdkApp(SleeperInternalCdkApp.DEMONSTRATION)
+                    .cdkApp(SleeperInternalCdkApp.STANDARD)


This is setting the wrong CDK app.

patchwork01 · 2026-07-31T09:48:04Z

                    .deployInstanceConfiguration(config)
-                    .cdkApp(SleeperInternalCdkApp.DEMONSTRATION)
+                    .cdkApp(SleeperInternalCdkApp.STANDARD)
+                    .configDir(configurationPath)


This is not correct. The code above is deriving the instance configuration from templates, but this is pointing to the original configuration file instead. I think this script still needs to write the files to a generated directory and point to that.

patchwork01

It looks like the demonstration deployment is broken right now, and I'm struggling to review the tests.

ca61688 and others added 14 commits July 7, 2026 09:02

6593: Add CommandArguments to DeployNewInstance

18fbcfe

6593: Make deployNew script use new DeployNewInstance file

cc33173

6593: Move uses of DeployNewIsntance to new Class

c74b7d4

6593: Replace old deployNewInstance with new version

e6b6070

6593: Add ignore table files flag

e11edbb

6593: Update deploy new instance help text to include ignoreTableFile…

e348a00

…s flag

7555 Set version number to 0.37.0

38a7c2c

7555 Performance figures for 0.37.0

fb3da30

7553 Changelog for 0.37.0

474d46d

7553 Adjust release summary

c62e6e0

7553 Note bug fix for DataFusion calls

29942a7

6593: Create initial unit test for deploy new instance

80a79c4

7605 Set version number to 0.37.1-SNAPSHOT

5eea52f

6593: Improve usage of DeployNewInstance constructor

a1411a6

ca61688 force-pushed the 6593-improve-usability-of-deploy-new-script branch from 227e99c to a1411a6 Compare July 7, 2026 08:03

ca61688 added 15 commits July 7, 2026 09:47

6593: Improve DeployNewInstance argument validation

584ed75

6593: Add remaining unit tests for DeployNewInstanceTest

e7a4210

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

0262eb7

6593: Add inital framework for DeployNewInstanceIT

e48525f

6593: Begin to unpick file reader changes

7593e6d

6593: Make DeployNewInstanceTest use tempDir

3925abc

6593: Remove unused dependency

972747f

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

1ca79b6

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

876c869

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

9b3b08c

6593: Have DeployNewInstance save update instance properties file

ac6c85b

Merge branch '6593-improve-usability-of-deploy-new-script' of github.…

3b3d60b

…com:gchq/sleeper into 6593-improve-usability-of-deploy-new-script

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

1300e94

6593: Remove old DeployNewInstance constructor

7d3f05b

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

c10561b

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

ae9ff7e

ca61688 marked this pull request as ready for review July 14, 2026 15:38

ca61688 added the needs-reviewer Pull requests that need a reviewer to be assigned label Jul 14, 2026

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

ec9ba16

patchwork01 reviewed Jul 15, 2026

View reviewed changes

patchwork01 assigned ca61688 Jul 15, 2026

patchwork01 removed the needs-reviewer Pull requests that need a reviewer to be assigned label Jul 15, 2026

ca61688 added 15 commits July 20, 2026 13:40

6593: Add check in DeployInstanceRequest constructor for propertiesFi…

3c26600

…le or confgDir being set

6593: Move clear directories method into FilesUtil and allow SaveLoca…

8189067

…lProperties to create directories

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

b8b722f

6593: Build SleeperInstanceConfiguration once in DeployExistingInstance

dcb9f88

6593: Assert on full DeployNewInstanceRequest

158a54e

6593: Add builder for DeployNewInstance class

203c375

6593: Remove unused variable

2c03d43

6593: Seperate InstancePropertiesLoader from FileReader

26c7fa3

6593: Make FileReader readFile default method

5e46c40

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

bbea9da

6593: Remove id,vpcId and subnet from DeployNewInstanceIT instance pr…

640411d

…operties

6593: Remove propertiesFile and configDir from DeployInstanceRequest

51e50cc

6593: Pass network parameters to CDK cml instead of resaving congif file

5110eb6

6593: Put FileReader and InstancePropertiesLoader back in AddTableCLient

a08283f

Merge branch 'develop' into 6593-improve-usability-of-deploy-new-script

d82ce18

rtjd6554 reviewed Jul 23, 2026

View reviewed changes

rtjd6554 approved these changes Jul 23, 2026

View reviewed changes

rtjd6554 assigned patchwork01 and unassigned ca61688 Jul 29, 2026

patchwork01 reviewed Jul 31, 2026

View reviewed changes

patchwork01 assigned ca61688 and unassigned patchwork01 Jul 31, 2026

patchwork01 requested changes Jul 31, 2026

View reviewed changes

Uh oh!

Conversation

ca61688 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Tests

Documentation

Uh oh!

patchwork01 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patchwork01 Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patchwork01 Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patchwork01 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ca61688 commented Jul 6, 2026 •

edited

Loading

patchwork01 left a comment •

edited

Loading

patchwork01 Jul 31, 2026 •

edited

Loading

patchwork01 Jul 31, 2026 •

edited

Loading