Implement Manifest V3 objects and deployment #1167

schulzh · 2022-09-16T10:09:41Z

Implements:

objects and logic required for reading and writing CC API V3 manifests
client interface and reactor implementation to apply the manifest
resource matching for v3 packages API
pushManifestV3 operation, similar to the existing pushManifest operation

I hope that the pushManifestV3 operation is complete (especially the "app readiness" checks), since I did not find a documentation of this process and had to reverse engineer it from the CLI.

I chose to pass a raw byte array as manifest in the client instead of an object, because that would've required to add the snakeyaml dependency to the client as well. Since the manifest is just passed to and applied by the CC anyway, I saw no benefit of passing as an object.

Unfortunately, I could not get a local CF instance set up for the integration tests. Due to licensing issues at my company I cannot use Virtualbox and setting it up on a local OpenStack "DevStack" instance wasn't working either. If requested, I can add integration tests, but I cannot ensure that they are working as intended.

Solves #999 and #1142 but not #1103 yet

This commit adds support for maintenance info when updating service instances Signed-off-by: Paul Harris <harrisp@vmware.com>

Signed-off-by: Paul Harris <harrisp@vmware.com>

This commit adds integration tests for Audit Events, and handles issues uncovered by the tests. It also adds some missing javadoc, and tweaks some naming and visibilities. [resolves cloudfoundry#1014] Signed-off-by: Paul Harris <harrisp@vmware.com>

Earlier versions of the client would automatically refresh an expired refresh token using the credentials it already had. This functionality was lost during the transition to newer versions of reactor. This commit reinstates that functionality. [resolves cloudfoundry#1051] Signed-off-by: Paul Harris <harrisp@vmware.com>

Adds fields required for compatibility with PCF 2.9, and update a potentially fragile test Signed-off-by: Paul Harris <harrisp@vmware.com>

Previously the audit events space and organization relationships were defined as custom objects, when they should have used the standard Relationships object. This commit corrects that error. Signed-off-by: Paul Harris <harrisp@vmware.com>

This commit adds logging of the X-Vcap-Request-Id header on responses to aid diagnosing communication issues with a cloud foundry server. The added information is available by enabling TRACE logging, minimally on cloudfoundry-client.response. [resolves cloudfoundry#981] Signed-off-by: Paul Harris <harrisp@vmware.com>

The project uses two word lists to generate random routes. These words have zero meaning within the project, they are effectively random strings that can be pronounced. Because they have no meaning to the project there is no reason for any even theoretically objectionable word to appear in the lists. Therefore this commit removes a non-exhaustive selection of such words. Signed-off-by: Paul Harris <harrisp@vmware.com>

This commit adds support for maintenance info at the operations layer, allowing service instances to be upgraded to new versions of a service. Signed-off-by: Paul Harris <harrisp@vmware.com> [resolves cloudfoundry#1050] [resolves cloudfoundry#1052]

) * Add variable substitution in ApplicationManifestUtils * Sanitize regex input * Throw exception on missing variable substitution * Add test for variable substitution in manifest

…ry#1157)

…stance to complete (cloudfoundry#1161) The update test does two things: 1.) it creates a managed service instance and 2.) it updates the managed service instance. The second part was failing because CloudController will refuse to update a service that's still being created. The update job was calling a method `waitForCompletionOnCreate` which was looking at the CreateServiceInstanceResponse object to see if it had an job id, if it did, then it would call `JobUtils.waitForCompletion(cloudFoundryClient, Duration.ofMinutes(1), createServiceInstanceResponse.getJobId().get())` and then return `getServiceInstanceIdByName(cloudFoundryClient, serviceInstanceName)`. The problem is that the call to `JobUtils.waitForCompletion` was creating a flow but nothing was subscribing to it, so it never actually ran and the test would not pause until the creation finished. The test has been modified such that it assumes the creation request will be asynch and return a job id. If it does not for some reason, the test would fail as wel call `.get()` on an optional object and calling that if the optional is empty will generate a NoSuchElementException. Then with the jobId, we call `JobUtils.waitForCompletion` wrapped in a flatMap, which ensures it's part of the overall flow which is being subscribed to. The same change was made after the update request, which also returns an asynch response and needs to be polled. Signed-off-by: Daniel Mikusa <dmikusa@vmware.com>

There are presently three LifecycleTypes supported: buildpack, docker and kpack. The first two have been supported, and this PR adds support for the third. It also includes a KpackData class for the lifecycle data that the kpack lifecycle type can return. API documentation clarifies what is allowed: http://v3-apidocs.cloudfoundry.org/version/3.122.0/index.html#lifecycles Includes a test which expands and deserializes the Kpack LifecycleType. Resolves cloudfoundry#1094 Signed-off-by: Daniel Mikusa <dmikusa@vmware.com>

Fix service credential binding details response

linux-foundation-easycla · 2022-09-16T10:09:45Z

The committers listed above are authorized under a signed CLA.

✅ login: schulzh / name: Hans Schulz (e883106)

pivotal-david-osullivan · 2022-09-16T13:23:55Z

Thanks for the PR @schulzh! Are you able to follow the link in the EasyCLA comment above to get the commit authorised
✅ and satisfy the bot?

SInce this is a significant addition, if you can add the integration tests I am happy to run them in a lab!

schulzh · 2022-09-16T13:38:52Z

I have already done the EasyCLA process and I'm waiting on the approval of my organization. I'll add integration tests then, thanks!

schulzh · 2022-09-21T13:43:53Z

Hi @pivotal-david-osullivan,

I've added integration tests. Could you please run them in your lab?
I've also completed the CLA. From my side this PR should be ready to merge, so I'd be happy to have it reviewed.

mheath

This is an amazing contribution. Thank you thank you thank you for all your hard work.

There's a fix that needs to happen in v3/SpacesTest.java (see the comments there)

Other than that, most of the rest of my comments are nitpicks about method order.

Once you fix these things, I'll merge this into its own branch so that I can write acceptance tests for it. Once we have tests, we'll merge this into main.

cloudfoundry-client/src/main/java/org/cloudfoundry/client/v3/applications/ApplicationsV3.java

...dfoundry-client/src/main/java/org/cloudfoundry/client/v3/resourcematch/_MatchedResource.java

cloudfoundry-client/src/main/java/org/cloudfoundry/client/v3/spaces/SpacesV3.java

cloudfoundry-client/src/main/java/org/cloudfoundry/client/v3/spaces/_ApplyManifestRequest.java

cloudfoundry-client/src/main/java/org/cloudfoundry/client/v3/spaces/_ApplyManifestResponse.java

...y-operations/src/main/java/org/cloudfoundry/operations/applications/DefaultApplications.java

mheath · 2022-10-05T18:38:16Z

integration-test/src/test/java/org/cloudfoundry/client/v3/ResourceMatchTest.java

+    private Mono<String> spaceId;
+
+    //TODO how to check if resource matching is enabled on this CF instance?
+    @IfCloudFoundryVersion(greaterThanOrEqualTo = CloudFoundryVersion.UNSPECIFIED) //TODO how to select this version?


I will look into this

integration-test/src/test/java/org/cloudfoundry/client/v3/SpacesTest.java

schulzh · 2022-10-06T13:31:20Z

Hi @mheath ,

Thank you for the review and kind words 🙂

I've addressed the issues, please take a look.

mheath · 2022-10-21T20:05:01Z

Will you change the branch from main to manifest-v3 on this PR and I will get it merged and then merge in my acceptance tests?

schulzh · 2022-10-22T00:14:34Z

Hi @mheath,

I've changed the base branch. It looks like its lagging behind main a few hounded commits. Should I rebase it?

schulzh · 2023-01-17T10:23:43Z

Hello @mheath,

do you have any updates on this? What is missing to get it merged to master?

twoseat and others added 30 commits May 28, 2020 10:52

Merge branch '3.x'

f7b5952

v4.8.0.BUILD-SNAPSHOT Development

272094c

Increase Support for Maintenance Info

0d134c0

This commit adds support for maintenance info when updating service instances Signed-off-by: Paul Harris <harrisp@vmware.com>

Merge branch '1048-maintenance-info' into 3.x

6a8affd

Merge branch '3.x'

9fb5485

Add support for events v3

f37e811

Polishing

8900835

Signed-off-by: Paul Harris <harrisp@vmware.com>

Advanced Polishing

e5a6c86

This commit adds integration tests for Audit Events, and handles issues uncovered by the tests. It also adds some missing javadoc, and tweaks some naming and visibilities. [resolves cloudfoundry#1014] Signed-off-by: Paul Harris <harrisp@vmware.com>

Merge branch '1014-v3-events' into 3.x

6ae68da

Merge branch '3.x'

f3a0924

Merge branch '1051-refresh-tokens' into 3.x

0cea1e0

Merge branch '3.x'

c8b85cc

Compatibility Updates

8c04919

Adds fields required for compatibility with PCF 2.9, and update a potentially fragile test Signed-off-by: Paul Harris <harrisp@vmware.com>

Merge branch 'compatibility-updates' into 3.x

2401346

Merge branch '3.x'

2dd0120

Correct Audit Events Relationships.

b92f9b6

Previously the audit events space and organization relationships were defined as custom objects, when they should have used the standard Relationships object. This commit corrects that error. Signed-off-by: Paul Harris <harrisp@vmware.com>

Merge branch 'audit-relationships' into 3.x

e6afdfb

Merge branch '3.x'

138c3d9

Merge branch '981-request-logging' into 3.x

e787600

Merge branch '3.x'

f6121c0

Merge branch 'word-updates' into 3.x

a6475cb

Merge branch '3.x'

0049f61

Merge branch '1050-maintenance-info-ops' into 3.x

097eb72

Merge branch '3.x'

0b299d2

v3.26.0.BUILD-SNAPSHOT Development

0d8fdf2

Merge branch '3.x'

64078f8

luschmar and others added 9 commits June 1, 2022 12:16

Add variable substitution in ApplicationManifestUtils (cloudfoundry#1153

f834b53

) * Add variable substitution in ApplicationManifestUtils * Sanitize regex input * Throw exception on missing variable substitution * Add test for variable substitution in manifest

Allow null values in map of service parameters (cloudfoundry#1159)

4f63c5f

Implement service instances and plans creation and update (cloudfound…

72b6b6b

…ry#1157)

Re-implement V3 Service Bindings (cloudfoundry#1158)

a9b80c9

v5.10.0.BUILD-SNAPSHOT Development

003de15

Fix service credential binding details response

2772431

Merge pull request cloudfoundry#1166 from radito3/main

dcba939

Fix service credential binding details response

schulzh force-pushed the manifest-v3 branch from ed6b1ec to 0649edc Compare September 16, 2022 10:12

Implement Manifest V3 objects and deployment

e883106

schulzh force-pushed the manifest-v3 branch from 0649edc to e883106 Compare September 16, 2022 10:20

schulzh added 2 commits September 21, 2022 15:36

add integration tests

c45ab19

some cleanup

92cbd00

pivotal-david-osullivan added triaged Initial triage of issue has been performed enhancement Improvement or new functionality labels Sep 23, 2022

Fix processes with 0 Instances not getting ready

6346e75

mheath suggested changes Oct 5, 2022

View reviewed changes

Apply review feedback

944a8a5

schulzh force-pushed the manifest-v3 branch from 1eab6ac to 944a8a5 Compare October 6, 2022 13:15

schulzh changed the base branch from main to manifest-v3 October 21, 2022 21:31

mheath merged commit 81f8eb9 into cloudfoundry:manifest-v3 Nov 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Manifest V3 objects and deployment #1167

Implement Manifest V3 objects and deployment #1167

schulzh commented Sep 16, 2022 •

edited

linux-foundation-easycla bot commented Sep 16, 2022 •

edited

pivotal-david-osullivan commented Sep 16, 2022

schulzh commented Sep 16, 2022

schulzh commented Sep 21, 2022

mheath left a comment

mheath Oct 5, 2022

schulzh commented Oct 6, 2022

mheath commented Oct 21, 2022

schulzh commented Oct 22, 2022

schulzh commented Jan 17, 2023

Implement Manifest V3 objects and deployment #1167

Implement Manifest V3 objects and deployment #1167

Conversation

schulzh commented Sep 16, 2022 • edited

linux-foundation-easycla bot commented Sep 16, 2022 • edited

pivotal-david-osullivan commented Sep 16, 2022

schulzh commented Sep 16, 2022

schulzh commented Sep 21, 2022

mheath left a comment

Choose a reason for hiding this comment

mheath Oct 5, 2022

Choose a reason for hiding this comment

schulzh commented Oct 6, 2022

mheath commented Oct 21, 2022

schulzh commented Oct 22, 2022

schulzh commented Jan 17, 2023

schulzh commented Sep 16, 2022 •

edited

linux-foundation-easycla bot commented Sep 16, 2022 •

edited