Encapsulate warnings in generalized node::Warnings and remove globals #30058

stickies-v · 2024-05-07T22:32:44Z

This PR:

moves warnings from common to the node library and into the node namespace (as suggested in rpc: return warnings as an array instead of just a single one #29845 (comment))
generalizes the warnings interface to Warnings::Set() and Warnings::Unset() methods, instead of having a separate function and globals for each warning. As a result, this simplifies the kernel::Notifications interface.
removes warnings.cpp from the kernel library
removes warning globals
adds testing for the warning logic

Behaviour change introduced:

the -alertnotify command is executed for all KernelNotifications::warningSet calls, which now also covers the large-work-invalid-chain warning
the GUI is updated automatically whenever a warning is (un)set, covering some code paths where it previously wouldn't be, e.g. when node::AbortNode() is called, or for the large-work-invalid-chain warning

Some discussion points:

is const std::string& id the best way to refer to warnings? Enums are an obvious alternative, but since we need to define warnings across libraries, strings seem like a straightforward solution.

DrahtBot · 2024-05-07T22:32:46Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	TheCharlatan
Stale ACK	ryanofsky

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30157 (Fee Estimation via Fee rate Forecasters by ismaelsadeeq)
#30141 (kernel: De-globalize validation caches by TheCharlatan)
#30110 (refactor: TxDownloadManager by glozow)
#29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)
#29015 (kernel: Streamline util library by ryanofsky)
#28830 ([refactor] Check CTxMemPool options in ctor by TheCharlatan)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot · 2024-05-07T22:39:36Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/24704508399}

TheCharlatan · 2024-05-08T08:24:06Z

Concept ACK on removing the warnings globals.

stickies-v · 2024-05-08T20:08:42Z

Force-pushed to address compilation failure on non-macOS systems:

moved GetNodeWarnings() (in rpc/util.cpp) to node::GetWarningsForRpc() (in node/warnings.cpp). Since rpc/util.cpp is in common, this causes issues after warnings are moved to node. I don't love this approach, but it seemed like the least bad one - open to suggestions if anyone has any?
updated bitcoin-chainstate.cpp to the new kernel::Notifications interface

TheCharlatan · 2024-05-22T20:21:42Z

src/node/warnings.h

+class Warnings
+{
+    mutable Mutex m_mutex;
+    std::map<std::string, bilingual_str> m_warnings GUARDED_BY(m_mutex);


As an alternative: How about making this a set of enums whose variants encode the various errors? Then we can add a function to convert the enums to strings (the enums and the conversion function could live in the kernel namespace) and use that to populate the vector for GetMessages().

I have considered that approach, but see a couple of difficulties with it:

We have kernel warnings ("unknown-new-rules-activated", "large-work-invalid-chain") and node warnings ("clock-out-of-sync", "pre-release-test-build", "fatal-internal-error"). I would rather not define the node warnings in the kernel namespace. We could have have kernel::Warning and node::Warning enums (and potentially more in the future), and have a std::variant<kernel::Warning, node::Warning> GetAllWarnings() to address that.

not all warnings are compile-time constants, see e.g. the "unknown-new-rules-activated" warning: const bilingual_str warning = strprintf(_("Unknown new rules activated (versionbit %i)"), bit);, so I'm no sure a conversion function is desirable?

I gave the kernel::Warning and node::Warning enums approach a go because it would be nice to have the possible warnings enumerated, and it ended up being less overhead than I expected. I'm leaning towards updating this PR with that approach. What do you think?

stickies-v@e4ea7ab - PoC but compiles and tests pass, needs rebase, cleanup and doc updates etc

Thanks for testing this out! I forgot that the we'd need to reconstruct the versionbit before, sorry about that. I think I like this better. The reason I am a bit hesitant about keying by strings is that it makes them harder to discover for outside users and forces them to use a variable-size data type as a key for mapping them.

EDIT: Looking at the PR more, I now see the previous discussion is about id strings, not message strings. In that case I agree with TheCharlatan it is a littler better to use enum ids than string ids.

I think in the longer run the string/enum distinction should not matter too much here.

In general, requiring kernel applications to get warning information from warning, or warningSet/warningUnset callbacks is not a nice API. It would make more sense for the kernel to treat errors and warnings similarly, and return this information directly in function return values or output arguments, instead of indirect callbacks. We have PR #29642 and #29700 which change kernel code to bubble up errors, and another PR could be made to bubble up warnings.

With these changes, you should be able to look at any kernel API function, and know exactly what error and warning conditions it can trigger, and have that information returned to the caller. We shouldn't remain in the current situation where when you call a kernel function, there is no way to know what fatal errors, or flush errors, or warnings it might trigger, and the only way to get that information by handling indirect callbacks.

I also think when designing these API's, it would be good to distinguish between errors and warning conditions that are directly actionable by callers, and those that aren't. Conditions that are directly actionable are conditions where callers might add special cases to do things like scheduling a retry, or dropping a cache to free up resources, or starting an interaction with the user that goes beyond displaying a message string. If the only way an API caller can realistically handle a condition is by showing or logging a message string, I think returning a string is much better than returning an enum, because a string, unlike an enum, can provide context needed to understand and solve the problem. But if the condition is more directly actionable and could be handled with special case code, returning structured information instead of (or along with) a string could be better.

I've incorporated the id-as-enum approach in the latest force-push, thank you very much for both of your extensive feedback.

ryanofsky

Code review ACK 445e6de. It was a little unclear in some places what behavior was supposed to be changing, and I think that could be documented better. But overall the changes look good and seem very positive.

ryanofsky · 2024-05-23T16:01:25Z

src/Makefile.am

@@ -235,6 +235,7 @@ BITCOIN_CORE_H = \
  node/txreconciliation.h \
  node/utxo_snapshot.h \
  node/validation_cache_args.h \
+  node/warnings.h \


In commit "move-only: move warnings from common to node" (6681395)

s/RPc/RPC/ in commit message

Thanks, fixed.

ryanofsky · 2024-05-23T16:06:43Z

src/node/kernel_notifications.cpp

-    static bool fWarned = false;
-    node::SetMiscWarning(warning);
-    if (!fWarned) {
+    if (node::g_warnings.Set(id, warning)) {


In commit "introduce and use the generalized node::Warnings interface" (6152c20)

Would be good to have release notes for this change mentioning new -alertnotify behavior, since it now can be trigged multiple times and in new cases

Sounds good, I've added release notes to cover that. The way I read the startup option docstring ("Execute command when an alert is raised (%s in cmd is replaced by message)"), I think the new behaviour aligns better with how I'd expect this option to behave.

ryanofsky · 2024-05-23T16:13:07Z

src/node/warnings.cpp

+    LOCK(m_mutex);
+    std::vector<bilingual_str> messages;
+    messages.reserve(m_warnings.size());
+    for (const auto& [id, msg] : m_warnings) {


In commit "introduce and use the generalized node::Warnings interface" (6152c20)

I guess this change also affects the order of warnings. It seems like previous code was written to consistently put the pre-release warning first, followed by the miscellaneous warnings, the large-work chain warning, and the time offset warning last.

Changing this is probably fine, but it would be good to note the change in the commit message. Could also note in setWarning documentation that ID affects order warnings are shown in.

I guess this change also affects the order of warnings.

It does indeed. I hadn't really considered this, and I think I agree that the current behaviour is fine - but I'll think about it more and open to suggestions.

Changing this is probably fine, but it would be good to note the change in the commit message.

I've updated the commit message section on behaviour change to:

Introduces behaviour change: - the `-alertnotify` command is executed for all `KernelNotifications::warningSet` calls, which now also covers the `LARGE_WORK_INVALID_CHAIN` warning. - previously, warnings were returned based on a predetermined order, e.g. with the "pre-release test build" warning always first. This is no longer the case, and Warnings::GetMessages() will return messages sorted by the id of the warning.

Could also note in setWarning documentation that ID affects order warnings are shown in.

I don't think this belongs in setWarning, kernel users could implement this how they want, so I've updated the Warnings::GetMessages() docstring to:

/** Return potential problems detected by the node, sorted by the * warning_type id */

ryanofsky · 2024-05-23T16:38:41Z

src/node/warnings.cpp

 }
-
-void SetfLargeWorkInvalidChainFound(bool flag)
+bool Warnings::Set(const std::string& id, const bilingual_str& message)


In commit "introduce and use the generalized node::Warnings interface" (6152c20)

Not important, but since these arguments are being inserted in a map, it would probably be a little better to pass them by value instead of reference, and use std::move so they can be moved instead of copied.

Ah makes sense, I've updated that, thanks.

ryanofsky · 2024-05-23T16:54:18Z

src/node/warnings.cpp

@@ -31,12 +32,15 @@ bool Warnings::Set(const std::string& id, const bilingual_str& message)
 {
    LOCK(m_mutex);
    const auto& [_, inserted]{m_warnings.insert({id, message})};
+    if (inserted) uiInterface.NotifyAlertChanged();


In commit "node: update uiInterface whenever warnings updated" (396b261)

Can commit message be updated to say what this change in behavior here is? It looks like now this UI notification will be triggered if a fatal error happens, so not sure if that is good or bad. It might be possible to avoid this by moving the uiInterface call to node/kernel_notifications.cpp instead. That would be good for consistency since uiInterface is already accessed there, and it would be nice for this new class not to rely on a global variable.

Can commit message be updated to say what this change in behavior here is?

I've updated the commit message:

node: update uiInterface whenever warnings updated This commit introduces slight behaviour change. Previously, the GUI status bar would be updated for most warnings, namely UNKNOWN_NEW_RULES_ACTIVATED, CLOCK_OUT_OF_SYNC and PRE_RELEASE_TEST_BUILD, but not for LARGE_WORK_INVALID_CHAIN (and not for FATAL_INTERNAL_ERROR, but that is not really meaningful). Fix this by always updating the status bar when the warnings are changed.

It looks like now this UI notification will be triggered if a fatal error happens, so not sure if that is good or bad

I'm not super familiar with the GUI, but I think it's mostly irrelevant for the internal fatal error case, or any case where we shutdown the node? We're not creating any messageboxes etc, the uiInterface.NotifyAlertChanged() just updates the status bar, so it's not blocking.

It might be possible to avoid this by moving the uiInterface call to node/kernel_notifications.cpp instead. That would be good for consistency since uiInterface is already accessed there, and it would be nice for this new class not to rely on a global variable.

I agree that not having a global variable in Warnings would be better, and that's how I originally implemented it at first too. But I think the current approach is the most consistent? It seems undesirable that we would ever want to create a warning, and then not show it in the GUI until another - unrelated - warning is created that does trigger the GUI update. So, if we agree that we always want to update the GUI when the warnings are modified, then I think just doing it inside the Set() and Unset() methods is the most sensible approach?

Since rpc/util.cpp is in common, also move GetNodeWarnings() to node::GetWarningsForRPC()

stickies-v

Thank you for the review, @TheCharlatan and @ryanofsky .

In this force push, I've:

rebased to address merge conflict from rpc: avoid copying into UniValue #30115
incorporated @TheCharlatan's suggestion to use enum class instead of std::string for the warning identifiers
addressed @ryanofsky's comments:
- improved commit messages to describe the introduced behaviour change and improved a couple of docstrings
- added release notes to describe the new -alertnotify behaviour
- updated the Warnings::Set() signature to pass by value and use move semantics
removed an unnecessary include in kernel/warning.h

stickies-v · 2024-05-23T19:19:16Z

src/Makefile.am

@@ -235,6 +235,7 @@ BITCOIN_CORE_H = \
  node/txreconciliation.h \
  node/utxo_snapshot.h \
  node/validation_cache_args.h \
+  node/warnings.h \


Thanks, fixed.

stickies-v · 2024-05-23T19:37:57Z

src/node/kernel_notifications.cpp

-    static bool fWarned = false;
-    node::SetMiscWarning(warning);
-    if (!fWarned) {
+    if (node::g_warnings.Set(id, warning)) {


Sounds good, I've added release notes to cover that. The way I read the startup option docstring ("Execute command when an alert is raised (%s in cmd is replaced by message)"), I think the new behaviour aligns better with how I'd expect this option to behave.

stickies-v · 2024-05-23T19:39:01Z

src/node/warnings.cpp

 }
-
-void SetfLargeWorkInvalidChainFound(bool flag)
+bool Warnings::Set(const std::string& id, const bilingual_str& message)


Ah makes sense, I've updated that, thanks.

stickies-v · 2024-05-23T19:51:06Z

src/node/warnings.cpp

@@ -31,12 +32,15 @@ bool Warnings::Set(const std::string& id, const bilingual_str& message)
 {
    LOCK(m_mutex);
    const auto& [_, inserted]{m_warnings.insert({id, message})};
+    if (inserted) uiInterface.NotifyAlertChanged();


Can commit message be updated to say what this change in behavior here is?

I've updated the commit message:

node: update uiInterface whenever warnings updated This commit introduces slight behaviour change. Previously, the GUI status bar would be updated for most warnings, namely UNKNOWN_NEW_RULES_ACTIVATED, CLOCK_OUT_OF_SYNC and PRE_RELEASE_TEST_BUILD, but not for LARGE_WORK_INVALID_CHAIN (and not for FATAL_INTERNAL_ERROR, but that is not really meaningful). Fix this by always updating the status bar when the warnings are changed.

It looks like now this UI notification will be triggered if a fatal error happens, so not sure if that is good or bad

I'm not super familiar with the GUI, but I think it's mostly irrelevant for the internal fatal error case, or any case where we shutdown the node? We're not creating any messageboxes etc, the uiInterface.NotifyAlertChanged() just updates the status bar, so it's not blocking.

It might be possible to avoid this by moving the uiInterface call to node/kernel_notifications.cpp instead. That would be good for consistency since uiInterface is already accessed there, and it would be nice for this new class not to rely on a global variable.

I agree that not having a global variable in Warnings would be better, and that's how I originally implemented it at first too. But I think the current approach is the most consistent? It seems undesirable that we would ever want to create a warning, and then not show it in the GUI until another - unrelated - warning is created that does trigger the GUI update. So, if we agree that we always want to update the GUI when the warnings are modified, then I think just doing it inside the Set() and Unset() methods is the most sensible approach?

stickies-v · 2024-05-23T20:03:22Z

src/node/warnings.cpp

+    LOCK(m_mutex);
+    std::vector<bilingual_str> messages;
+    messages.reserve(m_warnings.size());
+    for (const auto& [id, msg] : m_warnings) {


I guess this change also affects the order of warnings.

It does indeed. I hadn't really considered this, and I think I agree that the current behaviour is fine - but I'll think about it more and open to suggestions.

Changing this is probably fine, but it would be good to note the change in the commit message.

I've updated the commit message section on behaviour change to:

Introduces behaviour change: - the `-alertnotify` command is executed for all `KernelNotifications::warningSet` calls, which now also covers the `LARGE_WORK_INVALID_CHAIN` warning. - previously, warnings were returned based on a predetermined order, e.g. with the "pre-release test build" warning always first. This is no longer the case, and Warnings::GetMessages() will return messages sorted by the id of the warning.

Could also note in setWarning documentation that ID affects order warnings are shown in.

I don't think this belongs in setWarning, kernel users could implement this how they want, so I've updated the Warnings::GetMessages() docstring to:

/** Return potential problems detected by the node, sorted by the * warning_type id */

stickies-v · 2024-05-23T20:40:42Z

src/node/warnings.h

+class Warnings
+{
+    mutable Mutex m_mutex;
+    std::map<std::string, bilingual_str> m_warnings GUARDED_BY(m_mutex);


I've incorporated the id-as-enum approach in the latest force-push, thank you very much for both of your extensive feedback.

DrahtBot · 2024-05-24T05:39:56Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/25352180163}

Instead of having separate warning functions (and globals) for each different warning that can be raised, encapsulate this logic into a single class and allow to (un)set any number of warnings. Introduces behaviour change: - the `-alertnotify` command is executed for all `KernelNotifications::warningSet` calls, which now also covers the `LARGE_WORK_INVALID_CHAIN` warning. - previously, warnings were returned based on a predetermined order, e.g. with the "pre-release test build" warning always first. This is no longer the case, and Warnings::GetMessages() will return messages sorted by the id of the warning. Removes warnings.cpp from kernel.

This commit introduces slight behaviour change. Previously, the GUI status bar would be updated for most warnings, namely UNKNOWN_NEW_RULES_ACTIVATED, CLOCK_OUT_OF_SYNC and PRE_RELEASE_TEST_BUILD, but not for LARGE_WORK_INVALID_CHAIN (and not for FATAL_INTERNAL_ERROR, but that is not really meaningful). Fix this by always updating the status bar when the warnings are changed.

stickies-v · 2024-05-24T10:31:26Z

Oops, forgot to update bitcoin-chainstate.cpp again. Force pushed to fix that, and also slightly improved function signatures to use (kernel::Warning id, const bilingual_str& message) instead of (kernel::Warning id, const bilingual_str& warning).

stickies-v mentioned this pull request May 7, 2024

rpc: return warnings as an array instead of just a single one #29845

Merged

stickies-v force-pushed the 2024-04/move-warnings-node branch from cd8b420 to 4f06da2 Compare May 7, 2024 22:39

DrahtBot added the CI failed label May 7, 2024

This was referenced May 8, 2024

scripted-diff: Use LogInfo/LogDebug over LogPrintf/LogPrint #29641

Draft

Stratum v2 Template Provider (take 3) #29432

Draft

Broadcast own transactions only via short-lived Tor or I2P connections #29415

Open

This was referenced May 8, 2024

versionbits refactoring #29039

Open

[refactor] Check CTxMemPool options in ctor #28830

Open

stickies-v force-pushed the 2024-04/move-warnings-node branch 2 times, most recently from 3b2b72f to 445e6de Compare May 8, 2024 20:05

DrahtBot removed the CI failed label May 8, 2024

TheCharlatan reviewed May 22, 2024

View reviewed changes

This was referenced May 23, 2024

Fee Estimation via Fee rate Forecasters #30157

Draft

kernel: De-globalize validation caches #30141

Open

rpc: avoid copying into UniValue #30115

Merged

refactor: TxDownloadManager #30110

Draft

DrahtBot added the Needs rebase label May 23, 2024

ryanofsky reviewed May 23, 2024

View reviewed changes

DrahtBot requested a review from TheCharlatan May 23, 2024 17:10

stickies-v added 2 commits May 23, 2024 18:51

refactor: remove unnecessary AppendWarning helper function

5b377e0

move-only: move warnings from common to node

1a2e5e7

Since rpc/util.cpp is in common, also move GetNodeWarnings() to node::GetWarningsForRPC()

stickies-v force-pushed the 2024-04/move-warnings-node branch from 445e6de to 674da14 Compare May 23, 2024 20:52

DrahtBot removed the Needs rebase label May 23, 2024

stickies-v force-pushed the 2024-04/move-warnings-node branch from 674da14 to 5dfbc35 Compare May 23, 2024 21:11

stickies-v commented May 23, 2024

View reviewed changes

DrahtBot added the CI failed label May 24, 2024

stickies-v force-pushed the 2024-04/move-warnings-node branch from 5dfbc35 to 66d4f31 Compare May 24, 2024 08:28

stickies-v added 3 commits May 24, 2024 10:24

refactor: remove warnings globals

51cb837

stickies-v force-pushed the 2024-04/move-warnings-node branch from 66d4f31 to 51cb837 Compare May 24, 2024 09:24

DrahtBot removed the CI failed label May 24, 2024

DrahtBot mentioned this pull request May 24, 2024

kernel: Streamline util library #29015

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encapsulate warnings in generalized node::Warnings and remove globals #30058

Encapsulate warnings in generalized node::Warnings and remove globals #30058

stickies-v commented May 7, 2024

DrahtBot commented May 7, 2024 •

edited

DrahtBot commented May 7, 2024

TheCharlatan commented May 8, 2024

stickies-v commented May 8, 2024

TheCharlatan May 22, 2024

stickies-v May 23, 2024

stickies-v May 23, 2024 •

edited

TheCharlatan May 23, 2024

ryanofsky May 23, 2024 •

edited

stickies-v May 23, 2024

ryanofsky left a comment

ryanofsky May 23, 2024

stickies-v May 23, 2024

ryanofsky May 23, 2024

stickies-v May 23, 2024

ryanofsky May 23, 2024

stickies-v May 23, 2024

ryanofsky May 23, 2024

stickies-v May 23, 2024

ryanofsky May 23, 2024

stickies-v May 23, 2024

stickies-v left a comment

stickies-v May 23, 2024

stickies-v May 23, 2024

stickies-v May 23, 2024

stickies-v May 23, 2024

stickies-v May 23, 2024

stickies-v May 23, 2024

DrahtBot commented May 24, 2024

stickies-v commented May 24, 2024

Encapsulate warnings in generalized node::Warnings and remove globals #30058

Are you sure you want to change the base?

Encapsulate warnings in generalized node::Warnings and remove globals #30058

Conversation

stickies-v commented May 7, 2024

DrahtBot commented May 7, 2024 • edited

Code Coverage

Reviews

Conflicts

DrahtBot commented May 7, 2024

TheCharlatan commented May 8, 2024

stickies-v commented May 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stickies-v May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky May 23, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stickies-v left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrahtBot commented May 24, 2024

stickies-v commented May 24, 2024

DrahtBot commented May 7, 2024 •

edited

stickies-v May 23, 2024 •

edited

ryanofsky May 23, 2024 •

edited