Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry #7092

Merged
merged 5 commits into from
Apr 19, 2023

Conversation

timeabarna
Copy link
Contributor

@timeabarna timeabarna commented Mar 28, 2023

Summary

Have a command registry export-all-flows that does the following:
List all the buckets, for each bucket, list all flows, for each flow, list all versions and export each version. All files should be landing in a target directory provided as an argument of the function.

Have a command registry import-all-flows that does the following:
It takes a directory as input (the one created by the export-all-flows command), and goes through the files to create the corresponding buckets, flows and flows versions.
The original author, bucket id and flow id need to be kept.

Use cases:
use case 1: NiFi 1 -> connecting to NiFi Registry 1, "re-initialising" an existing NiFi Registry, the NiFi Registry does not change, only its configuration.
export everything
change the NR flow definition repo backend from local FS to database (for example), or from git to database, etc
import everything
the existing NiFi instance should not see any change
use case 2: NiFi 1 -> Registry 1, NiFi 2 -> Registry 2, Disaster Recovery kind of thing between two different sites.
export everything from NR1
import everything into NR2
If the import into NR2 is adding new versions, then NiFi 2 should be able to update an existing PG to a newer version of the flow.

NIFI-11327

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using mvn clean install -P contrib-check
    • JDK 11
    • JDK 17

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@timeabarna
Copy link
Contributor Author

Hello @bbende,

Can you please review this PR?

Thanks in advance.

Copy link
Contributor

@exceptionfactory exceptionfactory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this feature @timeabarna. I noted several general questions regarding the naming of the new REST Resource and potential concerns related to reading everything into memory. See the individual comments for details.

@timeabarna
Copy link
Contributor Author

Hello @bbende and @exceptionfactory,

Thanks you very much for your review. I've updated the code based on your recommendations, improved serialisation and added storage location update.

Can you please check again?

Thanks in advance.

@timeabarna timeabarna requested a review from bbende April 18, 2023 10:33
@timeabarna
Copy link
Contributor Author

Thanks @bbende and @exceptionfactory for your help, I've updated the PR.

Copy link
Contributor

@bbende bbende left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest updates look good, this is very close.

… documentation, adding some more error handling
@timeabarna timeabarna requested a review from bbende April 19, 2023 12:01
Copy link
Contributor

@bbende bbende left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the updates, this looks good to me. Everything is working as described, going to merge.

@bbende bbende merged commit 3df8c9d into apache:main Apr 19, 2023
asfgit pushed a commit that referenced this pull request Apr 19, 2023
* NIFI-11327 Add Export/Import All - NiFi CLI - NiFi Registry
* NIFI-11327 Addressing review comment, updating storage location, improving serialization
* NIFI-11327 Addressing review comments, updating Rest endpoints and storage location handling
* NIFI-11327 Restoring EventType to its original version
* NIFI-11327 Addressing review comments changing skipExisting, updating documentation, adding some more error handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants