-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry #7092
Conversation
Hello @bbende, Can you please review this PR? Thanks in advance. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this feature @timeabarna. I noted several general questions regarding the naming of the new REST Resource and potential concerns related to reading everything into memory. See the individual comments for details.
...nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/BucketFlowResource.java
Outdated
Show resolved
Hide resolved
...t/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/CommandOption.java
Outdated
Show resolved
Hide resolved
...t/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/CommandOption.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ExportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ExportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
Hello @bbende and @exceptionfactory, Thanks you very much for your review. I've updated the code based on your recommendations, improved serialisation and added storage location update. Can you please check again? Thanks in advance. |
...nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/BucketFlowResource.java
Outdated
Show resolved
Hide resolved
...ore/nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/BucketResource.java
Outdated
Show resolved
Hide resolved
.../nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/service/ServiceFacade.java
Outdated
Show resolved
Hide resolved
...y-core/nifi-registry-provider-api/src/main/java/org/apache/nifi/registry/hook/EventType.java
Outdated
Show resolved
Hide resolved
...ry-core/nifi-registry-client/src/main/java/org/apache/nifi/registry/client/BucketClient.java
Outdated
Show resolved
Hide resolved
...gistry-web-api/src/main/java/org/apache/nifi/registry/web/service/StandardServiceFacade.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
…orage location handling
Thanks @bbende and @exceptionfactory for your help, I've updated the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Latest updates look good, this is very close.
...nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/BucketFlowResource.java
Show resolved
Hide resolved
...main/java/org/apache/nifi/toolkit/cli/impl/result/registry/VersionedFlowSnapshotsResult.java
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Outdated
Show resolved
Hide resolved
...cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/registry/flow/ImportAllFlows.java
Show resolved
Hide resolved
… documentation, adding some more error handling
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the updates, this looks good to me. Everything is working as described, going to merge.
* NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry * NIFI-11327 Addressing review comment, updating storage location, improving serialization * NIFI-11327 Addressing review comments, updating Rest endpoints and storage location handling * NIFI-11327 Restoring EventType to its original version * NIFI-11327 Addressing review comments changing skipExisting, updating documentation, adding some more error handling
Summary
Have a command registry export-all-flows that does the following:
List all the buckets, for each bucket, list all flows, for each flow, list all versions and export each version. All files should be landing in a target directory provided as an argument of the function.
Have a command registry import-all-flows that does the following:
It takes a directory as input (the one created by the export-all-flows command), and goes through the files to create the corresponding buckets, flows and flows versions.
The original author, bucket id and flow id need to be kept.
Use cases:
use case 1: NiFi 1 -> connecting to NiFi Registry 1, "re-initialising" an existing NiFi Registry, the NiFi Registry does not change, only its configuration.
export everything
change the NR flow definition repo backend from local FS to database (for example), or from git to database, etc
import everything
the existing NiFi instance should not see any change
use case 2: NiFi 1 -> Registry 1, NiFi 2 -> Registry 2, Disaster Recovery kind of thing between two different sites.
export everything from NR1
import everything into NR2
If the import into NR2 is adding new versions, then NiFi 2 should be able to update an existing PG to a newer version of the flow.
NIFI-11327
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000
NIFI-00000
Pull Request Formatting
main
branchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
mvn clean install -P contrib-check
Licensing
LICENSE
andNOTICE
filesDocumentation