Diagnose Resource

[shirady]

Explain the changes

1. Add the script that will be run by the job that was created in Diagnose Resource noobaa-operator#1072

For more information see Jira MCGI-143

Issues: Gap

1. In this PR we don't handle the case of AWS STS (MCGI-149).

Testing Instructions:

Note: to actually run the tests you'll need to add the changes from this PR in noobaa-operator.

1. Based on the instructions here - the steps of ‘Build images’ and ‘Deploy noobaa’.
   Note: nb is a alias that runs the local operator from build/_output/bin (alias created by devenv).
2. For backingstore, Run:
   nb backingstore create aws-s3 <backing-store-name>
nb diagnostics analyze backingstore <backing-store-name>
3. For namespacestore, Run:
   nb namespacestore create aws-s3 <namespace-store-name>
nb diagnostics analyze namespacestore <namespace-store-name>

• It creates the log files and then deletes the created job.

Testes types: aws-s3, azure-blob, s3-compatible (with noobaa system as a server, the endpoint was taken from the field InternalDNS under S3 Addresses.

• Doc added/updated
• Tests added

[nimrod-becker]

Have you looked at s3cat.js ?
Might be some things we can share between the two instead of dupping

[shirady]

Have you looked at s3cat.js ? Might be some things we can share between the two instead of dupping

@nimrod-becker, no I was not aware of it. I'll have a look at it, thanks!

[shirady]

Hi,
@nimrod-becker I looked at the file s3cat.js, and I'm not sure it is duplicated...
As we defined the tests that we wish to run as: list objects, head\read to one of the objects, and write (I added the delete operation so I would leave the written file). The file includes those operations (and many more) but it is not "tailored" as I will use it in the later stories - I want it to be simple and work the same way in other SDKs (azure and gcp).
It also passes the secret key and access key as arguments which I want to avoid.
Of course, if there is a function there that we want to reuse, we can (but anyway, it will be re-use only for aws-s3 sdk, and not for the other scripts gcp and azure).

[romayalon]

@shirady I feel that the correct way to do this is to create "recipes" for the different kinds of tests you need to do from outside the s3 job area (or other cloud provider areas), also it'll help in the future if we would like to extend and make more tests/analysis, as it is today it's not very flexible in my opinion.
s3cat.js already implements the s3 client creation and the needed functions head, list, upload, and download when the resource is s3. it'll of course not help you with GCP and Azure because it implements s3 functions. What I expected is to have gcpcat/ blobcat for other cloud providers. NBcat is another reason to use this method, although it needs to be improved.

[romayalon]

@shirady @dannyzaken @nimrod-becker

Putting here some points from my suggestion discussed with Shira -

1. The Analyze CLI can be a very strong tool if it'll be flexible - which means it will be able to receive new types of tests in the future. (currently, this PR handles access checks only). I find this kind of tool helpful not only for support people but for the dev team for debugging purposes as well. (for example, imagine a performance test job that runs with a single CLI call)
2. In order to keep this tool flexible, we need to decouple the "report" handling from the cloud provider's SDK calls as a start. This will reduce the repetitive "report" code of the next cloud providers analysis code or different tests we will hopefully create.
3. Different types of tests can be created by the same job yaml as created here https://github.com/noobaa/noobaa-operator/pull/1072/files but with different parameters
4. More test configurations might add more SDK calls, these can be simply added to the different cloud utils.
5. I still believe that with more tests and more SDK calls, this code will be almost the same as s3cat.js with minor changes, so adapting it now might reduce the repetitive code later.