Track if remote listing completed succesully when validating incremental deploy plan

[Mpdreamz]

Targetting an empty s3 bucket with deploy plan

$ dotnet run --project src/tooling/docs-assembler -- deploy plan --environment dev --s3-bucket-name "elastic-docs-test" --out out.plan

info ::e.d.t.f.foLoggerFilter:: Configuration source: Local
info ::e.d.t.f.foLoggerFilter:: Version: 0.66.2-canary.0.19+fe64c22156176621ce8ec8f6c666479e1f56e3be
info ::e.d.t.f.topwatchFilter:: deploy plan :: Starting...
info ::Program :: Remote listing completed: True
info ::Program :: Total files to delete: 0
info ::Program :: Total files to add: 26175
info ::Program :: Total files to update: 0
info ::Program :: Total files to skip: 0
info ::Program :: Total local source files: 26175
info ::Program :: Total remote source files: 0
info ::d.a.d.SyncPlanStrategy:: No files discovered in S3, assuming a clean bucket resetting delete threshold to `0.0' as our plan should not have ANY deletions
info ::Program :: Plan written to out.plan
::set-output name=plan-valid::true

Targetting non existent s3 bucket

❯ dotnet run --project src/tooling/docs-assembler -- deploy plan --environment dev --s3-bucket-name "elastic-docs-does-not-exist" --out out.plan

info ::e.d.t.f.foLoggerFilter:: Configuration source: Local
info ::e.d.t.f.foLoggerFilter:: Version: 0.66.2-canary.0.20+050473985a85acfea7e071fc472fd913de4bb179
info ::e.d.t.f.topwatchFilter:: deploy plan :: Starting...
error::e.d.t.d.Log :: Bucket does not exist, cannot list objects (:0)
info ::Program :: Remote listing completed: False
info ::Program :: Total files to delete: 0
info ::Program :: Total files to add: 26175
info ::Program :: Total files to update: 0
info ::Program :: Total files to skip: 0
info ::Program :: Total local source files: 26175
info ::Program :: Total remote source files: 0
error::d.a.d.SyncPlanStrategy:: Remote files were not read to completion, cannot validate deployment plan

::set-output name=plan-valid::false
error::e.d.t.d.Log :: Plan is invalid, PlanValidationResult { Valid = False, DeleteRatio = 1, DeleteThreshold = 0.2 }, delete ratio: 1, remote listing completed: False (out.plan:0)

    The following errors and warnings were found in the documentation

Error: Bucket does not exist, cannot list objects

Error: Plan is invalid, PlanValidationResult { Valid = False, DeleteRatio = 1, DeleteThreshold = 0.2 }, delete ratio: 1, remote listing completed: False
NOTE: out.plan

    2 Errors / 0 Warnings / 0 Hints

info ::e.d.t.f.topwatchFilter:: deploy plan :: Finished in '00:00:01.2331545'

[Mpdreamz]

@reakaleek I updated this PR and tested it locally against a

• non existent s3 bucket that should fail
• an empty one that should succeed.

(see updated PR description).

To do that I am calling GetBucketAcl which has the following notes:

Do you think this requires updating the access rights of our deploy service principals? I don't think so but still digging for the references.

This operation is not supported for directory buckets. This implementation of the GET action uses the acl subresource to return the access control list (ACL) of a bucket. To use GET to return the ACL of the bucket, you must have the READ_ACP access to the bucket. If READ_ACP permission is granted to the anonymous user, you can return the ACL of the bucket without using an authorization header.

When you use this API operation with an access point, provide the alias of the access point in place of the bucket name.

When you use this API operation with an Object Lambda access point, provide the alias of the Object Lambda access point in place of the bucket name. If the Object Lambda access point alias in a request is not valid, the error code InvalidAccessPointAliasError is returned. For more information about InvalidAccessPointAliasError, see List of Error Codes .

If your bucket uses the bucket owner enforced setting for S3 Object Ownership, requests to read ACLs are still supported and return the bucket-owner-full-control ACL with the owner being the account that created the bucket. For more information, see Controlling object ownership and disabling ACLs in the Amazon S3 User Guide.