NIFI-9892: Updated Azure storage related processors to adhere to NiFi… #5944

markap14 · 2022-04-07T20:08:31Z

… best practices and cleaned up code a bit. Fixed several integration tests.

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

Enables X functionality; fixes bug NIFI-YYYY.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
Have you written or updated unit tests to verify your changes?
Have you verified that the full build is successful on JDK 8?
Have you verified that the full build is successful on JDK 11?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

… best practices and cleaned up code a bit. Fixed several integration tests.

gresockj

Overall, great updates, @markap14! Using the processors felt much more internally consistent now.

I noted that while ListAzureBlobStorage does set the filename attribute, it doesn't set it to a value that can be directly used by FetchAzureBlobStorage. Instead, the azure.blobname contains what would be the usable value. For example, filename contained test.txt and azure.blobname contained resources/test.txt in my test, and it's the latter that is needed by FetchAzureBlobStorage. Additionally, path was set to ./, which I think is not consistent. What do you think about setting path to the value before the filename, and defaulting the FetchAzureBlobStorage Blob Name property to ${path}/${filename}? Not sure how this would work at the root level, however.

Same issue for the *_12 processors. I have not exercised the ADLS processors yet.

jfrazee · 2022-04-08T16:59:42Z

@markap14 I looked these through and they all look like good changes (which you probably don't need to be told). There's one thing I wanted to call out that I think might be relevant with changing the default to ${filename}. I think it will break a quick creation of List -> Fetch without a user having to edit Fetch. The gist is that azure.blobname and filename aren't the same. The former includes the full path and the latter is just the basename (or it was that way).

See #3906 (comment)

jfrazee · 2022-04-08T19:30:08Z

Overall, great updates, @markap14! Using the processors felt much more internally consistent now.

I noted that while ListAzureBlobStorage does set the filename attribute, it doesn't set it to a value that can be directly used by FetchAzureBlobStorage. Instead, the azure.blobname contains what would be the usable value. For example, filename contained test.txt and azure.blobname contained resources/test.txt in my test, and it's the latter that is needed by FetchAzureBlobStorage. Additionally, path was set to ./, which I think is not consistent. What do you think about setting path to the value before the filename, and defaulting the FetchAzureBlobStorage Blob Name property to ${path}/${filename}? Not sure how this would work at the root level, however.

Same issue for the *_12 processors. I have not exercised the ADLS processors yet.

Oops. Overlapping replies.

${path}/${filename} is not inconsistent with some of the other Fetch processors so I think it's a reasonable default. The behavior of ${path} is also weird enough to call broken so it'd seem ok to change.

markap14 · 2022-04-09T14:32:33Z

Great catches @jfrazee and @gresockj . You are both indeed right - I did overlook the necessity of including the path attribute. Will update the Fetch processors to use ${path}/${filename} and ensure that the List processors are properly populating the path attribute.

…tribute within List/Fetch/Delete/Put/Move Blob Storage Processors

gresockj

Thanks for the blob name / path changes, @markap14! The processors behave as I expect, now. @jfrazee any further input?

turcsanyip

@markap14 Thanks for the changes! The old Blob processors needed a facelift definitely. However, the properties in the ADLS and mostly in the Blob_v12 blob processors tried to follow a concept. Please find my comments below.

turcsanyip · 2022-05-17T19:01:31Z

...ocessors/src/main/java/org/apache/nifi/processors/azure/storage/utils/AzureStorageUtils.java

+        final boolean accountNameAndKeySet = !credentialsSet && !sasTokenSet && accountNameSet && accountKeySet;
+        final boolean accountNameAndSasTokenSet = !credentialsSet && !accountKeySet && accountNameSet && sasTokenSet;


Following the naming concept of onlyCredentialsServiceSet, these variables could be named onlyAccountNameAndKeySet and onlyAccountNameAndSasTokenSet.

turcsanyip · 2022-05-17T19:34:34Z

...e-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java

@@ -93,6 +92,21 @@ public class PutAzureBlobStorage extends AbstractAzureBlobProcessor {
                  "will fail if the container does not exist.")
            .build();

+    private static final List<PropertyDescriptor> properties = Collections.unmodifiableList(Arrays.asList(


Upper case names should be used for static final fields => PROPERTIES

turcsanyip · 2022-05-17T19:51:26Z

...e-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java

+        AzureBlobClientSideEncryptionUtils.CSE_KEY_TYPE,
+        AzureBlobClientSideEncryptionUtils.CSE_KEY_ID,
+        AzureBlobClientSideEncryptionUtils.CSE_SYMMETRIC_KEY_HEX


I would consider to move these properties up, just after CREATE_CONTAINER. They belong to the "upload process / data", similar to CREATE_CONTAINER.

turcsanyip · 2022-05-17T19:52:38Z

...rocessors/src/main/java/org/apache/nifi/processors/azure/storage/DeleteAzureBlobStorage.java

+        AzureStorageUtils.PROP_SAS_TOKEN,
+        AzureStorageUtils.ENDPOINT_SUFFIX,
+        AzureStorageUtils.PROXY_CONFIGURATION_SERVICE,
+        DELETE_SNAPSHOTS_OPTION));


I would consider to move this property up, just after BLOB because it is related to the deleted blob.

Agreed that it's related to the blob, but based on importance, it should be low.

turcsanyip · 2022-05-17T19:58:37Z

...-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java

+        AzureStorageUtils.CONTAINER,
+        LISTING_STRATEGY,
+        AbstractListProcessor.RECORD_WRITER,
+        AzureStorageUtils.STORAGE_CREDENTIALS_SERVICE,
+        AzureStorageUtils.ACCOUNT_NAME,
+        AzureStorageUtils.ACCOUNT_KEY,
+        AzureStorageUtils.PROP_SAS_TOKEN,
+        AzureStorageUtils.ENDPOINT_SUFFIX,
+        PROP_PREFIX,
+        AzureStorageUtils.PROXY_CONFIGURATION_SERVICE,
+        ListedEntityTracker.TRACKING_STATE_CACHE,
+        ListedEntityTracker.TRACKING_TIME_WINDOW,
+        ListedEntityTracker.INITIAL_LISTING_TARGET


I think property ordering could be improved further:

PROP_PREFIX: just after CONTAINER

ListedEntityTracker.*: following LISTING_STRATEGY

turcsanyip · 2022-05-17T20:34:00Z

...cessors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage_v12.java

+        CONTAINER,
+        LISTING_STRATEGY,
+        RECORD_WRITER,
+        STORAGE_CREDENTIALS_SERVICE,
+        BLOB_NAME_PREFIX,
+        TRACKING_STATE_CACHE,
+        TRACKING_TIME_WINDOW,
+        INITIAL_LISTING_TARGET


In my opinion, the original ordering was better.
The concept was:

credential

what to list (container and prefix)

listing strategy and its related properties

We can change the order of these groups (e.g. move the credential down, though I always set it first) but I would not split [container + prefix] and [listing strategy + related properties].

I can agree with keeping the Blob Name Prefix with container. I do think that should come before the credentials. I think I'd tend to put listing strategy before credentials but honestly can go either way. But will move the Blob Name Prefix up to colocate with container, as I regard them as both related to one another and of equal importance.

One more thing to note here: TRACKING_STATE_CACHE, TRACKING_TIME_WINDOW and INITIAL_LISTING_TARGET properties depend on LISTING_STRATEGY so I believe the best place for them would be immediately after LISTING_STRATEGY.

turcsanyip · 2022-05-17T20:47:14Z

...ocessors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage_v12.java

+        AzureStorageUtils.CONTAINER,
+        BLOB_NAME,
+        CREATE_CONTAINER,
+        STORAGE_CREDENTIALS_SERVICE


Similar to the List processor, the original concept was:

credential first

container

blob

I would rather not split CONTAINER + CREATE_CONTAINER.

In reviewing this now after a while, I would actually argue that it makes more sense to push the CREATE_CONTAINER lower in the list. When I decide I want to push data to Azure Blob Storage, the first thing I want to configure is the container to push it in and the name of the object. Then I want to configure authentication. Whether or not the container should be auto-created is of secondary importance.

turcsanyip · 2022-05-17T20:51:36Z

...essors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureBlobStorage_v12.java

+        CONTAINER,
+        BLOB_NAME,
+        STORAGE_CREDENTIALS_SERVICE,
+        RANGE_START,
+        RANGE_LENGTH


Similar to List / Put, the concept was:

credential first

container

blob (and related)

The credential property can be moved down but I would not split the blob and range properties because they belong together.

@turcsanyip that makes sense from a purely organizational "like things should go together" perspective. When a user goes to configure a processor, though, there are certain things that the user will definitely expect to configure. The first thing that comes to mind is "What data do I want to fetch?" That's configured via the container and blob name. Next is authentication. So that should be the first thing presented to the user. We want to present the most important configuration options first. After that, it makes sense to group things with similar importance by categories. So so RANGE_START should definitely be located with RANGE_LENGTH. But it doesn't really make sense put those before credentials because they are far less important (in terms of what the user will want to configure). And it doesn't make sense to me to configure the Container, then the Credentials, then the Blob, because the Container and Blob together represent "what's the data that I want?"

@markap14 indeed, moving STORAGE_CREDENTIALS_SERVICE down further, I suggested in the comment, was not a good idea.
Actually, when it was placed at the first position originally, I also followed the "What data do I want to fetch?" approach. In Azure the container name is not globally unique (unlike bucket name on S3) but the container exists within a Storage Account. So, for identifying the data, the user needs: Storage Account > Container > Blob. STORAGE_CREDENTIALS_SERVICE contains the Storage Account Name, that's why it was placed before Container (and also because the credential is a property to be configured always).

turcsanyip · 2022-05-17T21:02:49Z

...cessors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureDataLakeStorage.java

+        FILESYSTEM,
+        DIRECTORY,
+        ADLS_CREDENTIALS_SERVICE,
+        FILE_FILTER,
+        RECURSE_SUBDIRECTORIES,
+        PATH_FILTER,


I believe DIRECTORY and RECURSE_SUBDIRECTORIES belong together and should not be split. Also FILE_FILTER and PATH_FILTER.

turcsanyip · 2022-05-17T21:14:41Z

...essors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureDataLakeStorage.java

+        FILESYSTEM_WITH_DEFAULT,
+        DIRECTORY_WITH_DEFAULT,
+        FILE,
+        ADLS_CREDENTIALS_SERVICE,
+        RANGE_START,
+        RANGE_LENGTH,
+        NUM_RETRIES


Similar to FetchAzureBlobStorage_v12: FILE and RANGE_*properties should not be split, I believe.

github-actions · 2022-09-22T00:18:55Z

We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours.

NIFI-9892: Updated Azure storage related processors to adhere to NiFi…

0ccf9c3

… best practices and cleaned up code a bit. Fixed several integration tests.

gresockj requested changes Apr 8, 2022

View reviewed changes

NIFI-9892: Updated to ensure that we make proper use of the 'path' at…

b896d52

…tribute within List/Fetch/Delete/Put/Move Blob Storage Processors

gresockj approved these changes May 4, 2022

View reviewed changes

turcsanyip self-requested a review May 17, 2022 19:05

turcsanyip requested changes May 17, 2022

View reviewed changes

github-actions bot added the Stale label Sep 22, 2022

github-actions bot closed this Oct 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-9892: Updated Azure storage related processors to adhere to NiFi… #5944

NIFI-9892: Updated Azure storage related processors to adhere to NiFi… #5944

markap14 commented Apr 7, 2022

gresockj left a comment

jfrazee commented Apr 8, 2022

jfrazee commented Apr 8, 2022

markap14 commented Apr 9, 2022

gresockj left a comment

turcsanyip left a comment

turcsanyip May 17, 2022

turcsanyip May 17, 2022

turcsanyip May 17, 2022

turcsanyip May 17, 2022

markap14 May 18, 2022

turcsanyip May 17, 2022

turcsanyip May 17, 2022

markap14 May 18, 2022

turcsanyip May 24, 2022

turcsanyip May 17, 2022

markap14 May 18, 2022

turcsanyip May 17, 2022

markap14 May 18, 2022

turcsanyip May 19, 2022

turcsanyip May 17, 2022

turcsanyip May 17, 2022

github-actions bot commented Sep 22, 2022

		final boolean accountNameAndKeySet = !credentialsSet && !sasTokenSet && accountNameSet && accountKeySet;
		final boolean accountNameAndSasTokenSet = !credentialsSet && !accountKeySet && accountNameSet && sasTokenSet;

NIFI-9892: Updated Azure storage related processors to adhere to NiFi… #5944

NIFI-9892: Updated Azure storage related processors to adhere to NiFi… #5944

Conversation

markap14 commented Apr 7, 2022

Description of PR

For all changes:

For code changes:

For documentation related changes:

Note:

gresockj left a comment

Choose a reason for hiding this comment

jfrazee commented Apr 8, 2022

jfrazee commented Apr 8, 2022

markap14 commented Apr 9, 2022

gresockj left a comment

Choose a reason for hiding this comment

turcsanyip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 22, 2022