Skip to content

NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI#3660

Closed
rfellows wants to merge 25 commits intoapache:analytics-frameworkfrom
rfellows:NIFI-6568-predictiveBackPressureUI
Closed

NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI#3660
rfellows wants to merge 25 commits intoapache:analytics-frameworkfrom
rfellows:NIFI-6568-predictiveBackPressureUI

Conversation

@rfellows
Copy link
Contributor

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

  • Add multi-line tooltips with detail for connection queue back pressure graphics.
  • Add estimated time to back pressure to connections summary table.
  • Add back pressure prediction ticks.
  • add moment.js to format predicted time to back pressure
  • tweak summary table headings to match data displayed. re-order connection summary columns

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@ai-christianson
Copy link
Contributor

lgtm

@YolandaMDavis
Copy link
Contributor

@rfellows this looks good, paired this with #3663. Tested with growing, constant and emptied queues and UI reflected values as expected.

LGTM +1

@YolandaMDavis
Copy link
Contributor

@rfellows is it ok to merge this PR?

@rfellows
Copy link
Contributor Author

@mcgilman or @scottyaslan would you mind having a look at this PR?

ai-christianson and others added 23 commits August 27, 2019 12:42
(cherry picked from commit e013b91)

DFA-9 - updated logging and corrected logic for checking if not in backpressure

(cherry picked from commit a1f8e70)
(cherry picked from commit 050e0fc)

(cherry picked from commit 9fd365f)
…entAccess, updated interface to use connection id

(cherry picked from commit 14854ff)

DFA-9 - reduced snapshot interval to 1 minute

(cherry picked from commit 36abb0a)
… just use ConnectionStatusAnalytics directly
…aces as we can just use ConnectionStatusAnalytics directly"

This reverts commit 5b9fead.
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.

DFA-9 - added logging

* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval

* DFA-9 - checkstyle fixes
…lso changes to properly reflect when predictions can be made vs not.

(cherry picked from commit 6fae058)
(cherry picked from commit 6d7a13b)
…d variable names

(cherry picked from commit 58c7c81)
(cherry picked from commit b6e35ac)
Updates to support multiple variables for features, clearing cached regression model based on r-squared values

Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).

Ratcheck fixes

Added test for SimpleRegression. Minor fix for OLS model

fixed test errors

fixed checkstyle errors

(cherry picked from commit fab411b)
… object. Also allow configurable model from nifi.properties

NIFI-6566 - changes to allow scoring configurations for model in nifi.properties

NIFI-6566 - added default implementation value to NiFiProperties

NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes apache#3663
…ns. Added check in ConnectionStatusAnalytics to confirm expected model by type
@asfgit asfgit force-pushed the analytics-framework branch from e9986c2 to dbbd41b Compare August 27, 2019 16:42
…the UI

* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
@rfellows rfellows force-pushed the NIFI-6568-predictiveBackPressureUI branch from b743c4e to 4161e64 Compare August 27, 2019 18:52
@mcgilman
Copy link
Contributor

I'm happy to try to review the front end changes here...

…the connection summary table. Also added a js doc comment.
@rfellows
Copy link
Contributor Author

@mcgilman I have updated the PR with the minor changes as discussed.

@mcgilman
Copy link
Contributor

@rfellows Thanks! I am a +1 on the front end changes introduced here. I did not review the other commits.

@mcgilman mcgilman assigned mcgilman and unassigned mcgilman Aug 28, 2019
@mcgilman mcgilman self-requested a review August 28, 2019 19:38
@asfgit asfgit force-pushed the analytics-framework branch from daad09c to 9e35e4c Compare August 28, 2019 20:20
@ai-christianson
Copy link
Contributor

Merged. Pls close this ticket as commit hook does not work with feature branches.

@rfellows rfellows closed this Aug 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants