Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add <datadir>/settings.json persistent settings storage #15935

Draft
wants to merge 5 commits into
base: master
from

Conversation

@ryanofsky
Copy link
Contributor

ryanofsky commented May 1, 2019

This is based on #15934. The non-base commits are:


Persistent settings are used in followup PRs #15936 to unify gui settings between bitcoin-qt and bitcoind, and #15937 to add a load_on_startup flag to the loadwallet RPC and maintain a dynamic list of wallets that should be loaded on startup that also can be shared between bitcoind and bitcoin-qt.


This is marked worked-in-progress because it needs unit tests.

@DrahtBot

This comment has been minimized.

Copy link
Contributor

DrahtBot commented May 2, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #16278 (tests: Reduce compile-time memory usage, compilation time and unneccessary recompiles by removing unused includes in tests by practicalswift)
  • #16273 (refactor: Reduce compile-time memory usage by 2%, compilation time by 2% and avoid unnecessary recompiles by removing unused includes by practicalswift)
  • #16255 (util: Remove code to cache datadir by MarcoFalke)
  • #16224 (gui: Bilingual GUI error messages by hebasto)
  • #16115 (On bitcoind startup, write config args to debug.log by LarryRuane)
  • #16097 (WIP: Prevent meaningless negating of arguments by hebasto)
  • #16069 (test: move-only: Split large tests into smaller compile units by MarcoFalke)
  • #15934 (Separate settings merging from parsing by ryanofsky)
  • #14866 (Improve property evaluation way in bitcoin.conf by AkioNak)
  • #14045 (refactor: Fix the chainparamsbase -> util circular dependency by Empact)
  • #8994 (Testchains: Introduce custom chain whose constructor... by jtimon)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@luke-jr

This comment has been minimized.

Copy link
Member

luke-jr commented May 2, 2019

NACK, this is redundant with #11082 which is a better solution.

@ryanofsky ryanofsky force-pushed the ryanofsky:pr/rwset branch from 44532f8 to 0ec6245 May 2, 2019
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request May 3, 2019
This change makes it easier to add a new persistent settings source in followup
PR bitcoin#15935 without introducing new bugs and inconsistencies.

This PR does not change any behavior and has good test coverage through the
util_SettingsMerge test added in bitcoin#15869.

Right now settings from different sources are partially merged when settings
are parsed, and partially merged when settings are retrieved. The logic
implemented merging during retrieval is repeated 5 places:

- ArgsManagerHelper::GetArg
- ArgsManagerHelper::GetNetBoolArg
- ArgsManager::GetArgs
- ArgsManager::IsArgNegated
- ArgsManager::GetUnsuitableSectionOnlyArgs

which makes it cumbersome introduce a new settings source. Inconsistencies
between the 5 implementations are also responsible for some confusing and
counterintuitive behaviors (documented in this PR but not changed for now):

  - Ignored command line arguments (see ArgsManager::ParseParameters)
  - Inconsistently ignored network-specific arguments (see GetSetting)
  - Reappearing negated config arguments (see GetListSetting)

After this change, merging of settings happens in one place: a new
settings.h/settings.cpp file in code that has more comments and is less
duplicative. It separates merging from parsing, using a data structure that
holds complete representation of parsed config files and command line options,
instead of the current more abstract data structure that has negated values
represented by placeholder map entries.
@ryanofsky ryanofsky force-pushed the ryanofsky:pr/rwset branch from 0ec6245 to f6a9cf0 May 3, 2019
ryanofsky added 5 commits Mar 5, 2019
Implement merging of settings from different sources (command line and config
file) separately from parsing code in system.cpp, so it is easier to add new
sources.

Document current inconsistent merging behavior without changing it.

This commit only adds new settings code without using it. The next commit calls
the new code to replace existing code in system.cpp.
Get rid of settings merging code mixed in settings parsing code in
util/system.cpp repeated 5 places, inconsistently:

- ArgsManagerHelper::GetArg
- ArgsManagerHelper::GetNetBoolArg
- ArgsManager::GetArgs
- ArgsManager::IsArgNegated
- ArgsManager::GetUnsuitableSectionOnlyArgs

Having settings merging code separated from parsing simplifies parsing somewhat
(for example negated values can simply be represented as false values instead
of partially cleared or emply placeholder lists).

Having settings merge happen one place instead of 5 makes it easier to add new
settings sources and harder to introduce new inconsistencies in the way
settings are merged.

This commit does not change behavior in any way.
Persistent settings are used in followup PRs #15936 to unify gui settings
between bitcoin-qt and bitcoind, and #15937 to add a load_on_startup flag to
the loadwallet RPC and maintain a dynamic list of wallets that should be loaded
on startup that also can be shared between bitcoind and bitcoin-qt.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request May 7, 2019
This change makes it easier to add a new persistent settings source in followup
PR bitcoin#15935 without introducing new bugs and inconsistencies.

This PR does not change any behavior and has good test coverage through the
util_SettingsMerge test added in bitcoin#15869.

Right now settings from different sources are partially merged when settings
are parsed, and partially merged when settings are retrieved. The logic
implemented merging during retrieval is repeated 5 places:

- ArgsManagerHelper::GetArg
- ArgsManagerHelper::GetNetBoolArg
- ArgsManager::GetArgs
- ArgsManager::IsArgNegated
- ArgsManager::GetUnsuitableSectionOnlyArgs

which makes it cumbersome introduce a new settings source. Inconsistencies
between the 5 implementations are also responsible for some confusing and
counterintuitive behaviors (documented in this PR but not changed for now):

  - Ignored command line arguments (see ArgsManager::ParseParameters)
  - Inconsistently ignored network-specific arguments (see GetSetting)
  - Reappearing negated config arguments (see GetListSetting)

After this change, merging of settings happens in one place: a new
settings.h/settings.cpp file in code that has more comments and is less
duplicative. It separates merging from parsing, using a data structure that
holds complete representation of parsed config files and command line options,
instead of the current more abstract data structure that has negated values
represented by placeholder map entries.
@ryanofsky ryanofsky force-pushed the ryanofsky:pr/rwset branch from f6a9cf0 to 39005eb May 7, 2019
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request May 8, 2019
This change makes it easier to add a new persistent settings source in followup
PR bitcoin#15935 without introducing new bugs and inconsistencies.

This PR does not change any behavior and has good test coverage through the
util_SettingsMerge test added in bitcoin#15869.

Right now settings from different sources are partially merged when settings
are parsed, and partially merged when settings are retrieved. The logic
implemented merging during retrieval is repeated 5 places:

- ArgsManagerHelper::GetArg
- ArgsManagerHelper::GetNetBoolArg
- ArgsManager::GetArgs
- ArgsManager::IsArgNegated
- ArgsManager::GetUnsuitableSectionOnlyArgs

which makes it cumbersome introduce a new settings source. Inconsistencies
between the 5 implementations are also responsible for some confusing and
counterintuitive behaviors (documented in this PR but not changed for now):

  - Ignored command line arguments (see ArgsManager::ParseParameters)
  - Inconsistently ignored network-specific arguments (see GetSetting)
  - Reappearing negated config arguments (see GetListSetting)

After this change, merging of settings happens in one place: a new
settings.h/settings.cpp file in code that has more comments and is less
duplicative. It separates merging from parsing, using a data structure that
holds complete representation of parsed config files and command line options,
instead of the current more abstract data structure that has negated values
represented by placeholder map entries.
@ryanofsky ryanofsky force-pushed the ryanofsky:pr/rwset branch from 39005eb to 70675c3 May 8, 2019
@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented May 8, 2019

re: #15935 (comment)

Just as background, the settings file is not supposed to replace the config file. The settings file is supposed to replace some QSettings uses in bitcoin-qt, implementing type-safe persistent storage, and sharing this functionality with bitcoind, so settings modified at runtime will be stored in a common place regardless of whether you're using the GUI or a plain RPC interface. (You can see in 0ab41dd from #15936 how this works in the GUI or in 04c80c4 from #15937 how this can be used in an RPC.)

this is redundant with #11082 which is a better solution.

Every solution has some tradeoffs, and I'm sure you can elaborate more on advantages of #11082 compared to this. But here are what I see as nice things about this approach:

  • Standard format. The bitcoin.conf format works pretty well as a manual configuration mechanism. But the format is unspecified and unstable, and less appealing for use as persistent key/value store compared to JSON which is well understood and supported by many tools and libraries.

  • Uniformity of representation. Since bitcoin.conf files are meant to be written by people and parsed by software, it's reasonable for them to provide many ways to do things like represent true/false values (foo=1 foo= foo=23 foo=-1, nofoo=0 for true / foo=0 nofoo=1 foo=false for false), or to strip whitespace. But for a file that is meant to be round-trip read and written by software, only accepting plain true false keywords as bools and not mangling strings is a better way to avoid complexity, configuration errors, and software bugs.

  • Flexibility of representation. bitcoin.conf format can't currently represent strings with pound (#) or newline characters or strings with various types of whitespace. It can also only awkwardly represent lists by repeating settings (so there is no distinction between a non-list value and a single-element list). I think these things are likely to cause bugs, unnecessary limitations, headaches, and extra boilerplate in application-level code that reads and writes settings.

  • Maintainability. JSON for all that's good and bad about it is stable and usable enough and isn't going to change. Even writing this PR didn't require adding new parsing or formatting code. By contrast, #11082 not only adds new code right now to parse and generate bitcoin_rw.conf files, but will also require more work on an ongoing basis whenever the bitcoin.conf format is extended, because any new change will now require roundtrip support. (For example if we added an escaping mechanism it would require new code to escape values in addition to new code parsing escapes.)

This PR is also not mutually exclusive with #11082. I don't know exactly what features you wanted to implement on top of #11082, but if you are thinking of new features different from #15936 and #13937 that call for injecting settings into a human-edited config file, nothing in this PR should get in the way of that.

There were some other issues I had with #11082 when I reviewed it (adding a new bitcoin.conf parser instead of extending the existing one, and concerns about init order and inconsistent interpretation of network sections in bitcoin.conf and bitcoin_rw.conf), but those issues were specific to the implementation and not inherent to the approach.

@jonasschnelli

This comment has been minimized.

Copy link
Member

jonasschnelli commented May 18, 2019

Concept ACK.

Standard format. The bitcoin.conf format works pretty well as a manual configuration mechanism. But the format is unspecified and unstable, and less appealing for use as persistent key/value store compared to JSON which is well understood and supported by many tools and libraries.

I strongly agree with that.

@luke-jr

This comment has been minimized.

Copy link
Member

luke-jr commented May 18, 2019

INI is a standard format, supported by at least as many tools and libraries for longer. It has been working fine for bitcoin.conf and rwconf for years now.

JSON, on the other hand, is fragile, doesn't allow users to add comments, and is currently supposed to only be used for the JSON-RPC/REST interfaces of Core.

The only thing to gain over the current INI format would be to avoid rewriting the entire file for changes. JSON doesn't get us that (nor anything else).

@jnewbery

This comment has been minimized.

Copy link
Member

jnewbery commented Jul 2, 2019

Concept ACK

@ajtowns

This comment has been minimized.

Copy link
Contributor

ajtowns commented Jul 3, 2019

Concept NACK from me too for much the same reasons @luke-jr gives. JSON's fine for a human-readable wire format where programs in different languages need to exchange data (ie, RPC), but it's not a great configuration language at all. I don't think "the settings file is not supposed to replace the config file" will hold up -- @jnewbery's already suggesting this be used for user configuration in 16248, eg.

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 3, 2019

re: #15935 (comment)

I can make a more complete response later but I think the quote "the settings file is not supposed to replace the config file" is not being understood correctly. I'm using "config" there to refer to static configuration and "settings" to refer to dynamic configuration (based on bitcoin.conf, which is static, and QSettings, which is dynamic, in current code).

My thought is that more advanced command-line / linux type users will want to sit down and write out their configurations in text files with comments, but less advanced users will not be editing text files and will be happy if preferences they set through gui or rpc interfaces are just persisted in a simple format without support for comments. I can write more but @ajtowns, it would be helpful to me if you could say how you would want GUI and runtime settings to be stored ideally (for example, the list of wallets to load on startup).

@ajtowns

This comment has been minimized.

Copy link
Contributor

ajtowns commented Jul 3, 2019

@ryanofsky I think it would make sense to do any of:

  • just keep doing what we're already doing
  • switch to a different machine-readable format that can do key-value store but isn't tied to the gui (eg bdb, serialize.h, sqlite)
  • have a limited machine-updatable version of our human-readable config format (luke's PR roughly)
  • switch to a new format (eg toml, yaml, json5/hjson/hocon, etc) for our human-readable static config, and use the same format (or a restricted version of it) for the machine-modifiable dynamic/persistent config

If we stuck with a primarily machine-readable format, having a "dumpsettings" / "loadsettings" RPC pair that translated that to and from the bitcoin.conf ini-format could be workable for people who want to play with persistent settings outside of the GUI (eg, playing with settings in the gui to get something that seems to work right, then wanting to copy some of them to bitcoind on a different computer).

I don't think it's realistic to treat json as a "machine-readable format" for case 2 because it's both human readable and more powerful than our ini-format so it's an obvious hammer to pick up for static configuration cases where the ini-format is awkward, cf #16248 (comment) and #16248 (comment) . Maybe we want to make bitcoin.conf json (or a supserset of json), but that ought to be a separate discussion.

Hope that makes sense

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 3, 2019

@ajtowns, it sounds like your opinion is basically the same as Luke's, and a concrete thing maybe we all would be happy with would be a PR that stores persistent GUI and application settings in an INI file instead of a JSON file.

If I am understanding correctly, you are not any citing direct advantages to the INI format for storing dynamic settings, but instead you're making more of a slippery slope precautionary argument: that if we add a way to represent new settings in JSON form, it will encourage creation of ever more complex settings that can be conveniently represented as JSON lists and objects that will be easy to manipulate with GUIs and dynamic configuration, but will be impossible or awkward to represent in INI format. And therefore as a result, even though the settings.json file is intended to be only machine edited, users will end up modifying it anyway and wind up putting settings in a read-write file without comments, that would be more appropriately stored in a read-only file with comments. Let me know if I have this wrong, or if you do think INI format is a better format for dynamic configuration than the JSON format for other reasons that I have not understood.

But to follow up, would you be happy with a PR like this one puts settings in a bitcoin_rw.conf file with a big warning on top?

# bitcoin_rw.conf: Dynamic settings for Bitcoin Core data directory
#
# Warning: DO NOT EDIT this file. Settings in this file can be overridden at
# runtime and comments will be lost. Instead, to statically configure the
# bitcoin node and wallet, create a bitcoin.conf file with the desired settings.

proxy=host:123
prune=2861
wallet=wallet1
wallet=wallet2

instead of the current settings.json file:

{
    "proxy": "host:123",
    "prune": 2861,
    "wallet": ["wallet1", "wallet2"]
}

I will respond to other alternatives you suggested, but I think they are all have deficiencies

just keep doing what we're already doing

The status quo is that we're storing dynamic settings in the registry on windows, in
in $HOME/.config/Bitcoin/Bitcoin-Qt.conf files on linux and $HOME/Library/Preferences plist files on Mac OS, and that dynamic settings are not tied to the datadir, and not shared between bitcoind and bitcoin-qt. I think this is unfortunate and want to fix this problem before adding support for more dynamic settings.

switch to a different machine-readable format that can do key-value store but isn't tied to the gui (eg bdb, serialize.h, sqlite)

I'm not sure what advantages you see in this approach compared to the current PR, other than that the settings files would be less accessible, so advanced users would be more encouraged to create static configurations where appropriate. I don't mean to be unfair, but I think we could get the same benefit with less code by just tweaking the current PR and having it save settings.json.gz or settings.json.rot13 files that would also be a pain in the ass to edit.

have a limited machine-updatable version of our human-readable config format (luke's PR roughly)

I do think our current format is pretty terrible (stupid about bool, and lists, can't represent strings containing #, etc, see previous comments) and I have other specific issues with Luke's PR (again see previous comments). But I could live with this approach and I will attempt to implement it if this PR is dead or going nowhere.

switch to a new format (eg toml, yaml, json5/hjson/hocon, etc) for our human-readable static config, and use the same format (or a restricted version of it) for the machine-modifiable dynamic/persistent config

As far as I can see (please correct me if I'm mistaken) all of these formats have 0 advantages over plain JSON for dynamic configuration, and very few advantages over current INI format for static configuration, and would trigger lots of bikeshedding, and require new parsing code. I think the major advantage of this over PR all your suggested alternatives but especially this one, is that it's ridiculously simple and requires no new parsing code at all: 70675c3

@ajtowns

This comment has been minimized.

Copy link
Contributor

ajtowns commented Jul 3, 2019

@ajtowns, it sounds like your opinion is basically the same as Luke's, and a concrete thing maybe we all would be happy with would be a PR that stores persistent GUI and application settings in an INI file instead of a JSON file.

Yeah; the benefit of that approach that the json one doesn't have is that you can just cut and paste lines that the gui has created into your permanent bitcoin.conf directly.

But to follow up, would you be happy with a PR like this one puts settings in a bitcoin_rw.conf file with a big warning on top?

I think a warning makes sense.

The status quo is that we're storing dynamic settings in the registry on windows, in
in $HOME/.config/Bitcoin/Bitcoin-Qt.conf files on linux and $HOME/Library/Preferences plist files on Mac OS, and that dynamic settings are not tied to the datadir, and not shared between bitcoind and bitcoin-qt. I think this is unfortunate and want to fix this problem before adding support for more dynamic settings.

A different approach would be to say "bitcoin-qt has dynamic settings -- it has a GUI that lets you do dynamic things" and "bitcoind does not have dynamic settings", and decide that's the way it's meant to be. Not having dynamic settings means that if you "shutdown and restart" you're back to a clean slate which can be good, and could be appropriate for a gui-less daemon. If so, it could just be enough to have some way of viewing/exporting the current dynamic settings from the gui, so you can paste them into your bitcoin.conf. Not really advocating this, but I think it's actually workable.

switch to a different machine-readable format that can do key-value store but isn't tied to the gui (eg bdb, serialize.h, sqlite)

I'm not sure what advantages you see in this approach compared to the current PR, other than that the settings files would be less accessible, so advanced users would be more encouraged to create static configurations where appropriate.

It's more that if you've got "settings.json" or "bitcoin_rw.conf" sitting around, people are going to go "ah, great, I see what's going on" and edit it. If they see an sqlite file, they might fire up sqlite, but it'll have a schema so they'll only be able to touch it in fairly sensible ways. If they see a serialize.h or bdb file, they probably won't touch it, but if they do it'll be using tools that will probably get it right and likely won't be surprised if things break. With a .json (or .json.gz or .ini) they're going to go "oh cool! look what i found!" and fire up notepad and mess around, and you need to give detailed error messages to help them find their mistakes and manage expectations so they're not terribly surprised when their formatting/comments/ordering/additions disappear or otherwise get mangled.

Using sqlite would get the "avoid rewriting the entire file for changes" benefit for whatever that's worth.

have a limited machine-updatable version of our human-readable config format (luke's PR roughly)

I do think our current format is pretty terrible (stupid about bool, and lists, can't represent strings containing #, etc, see previous comments)

I don't disagree, but if that's the problem, let's actually work on it and improve the actual config format. But:

As far as I can see (please correct me if I'm mistaken) all of these formats [toml, yaml, json5/hjson/hocon, etc] have 0 advantages over plain JSON for dynamic configuration, and very few advantages over current INI format for static configuration, and would trigger lots of bikeshedding, and require new parsing code.

They'd all also require some special handling for upgrading from today's bitcoin.conf to whatever the future one is, which seems like a pain too. But if the conclusion there is "our custom ini stuff is what's best for bitcoin.conf" then I don't think it makes sense to complain about how terrible our current format is, if it's still the best we can do.

and I have other specific issues with Luke's PR (again see previous comments)

Sure. I suppose I'm obliged to review that PR now too...

I think the major advantage of this over PR all your suggested alternatives but especially this one, is that it's ridiculously simple and requires no new parsing code at all

If it's the wrong thing, it doesn't really matter how easy it is...

@jnewbery

This comment has been minimized.

Copy link
Member

jnewbery commented Jul 10, 2019

But I could live with this approach and I will attempt to implement it if this PR is dead or going nowhere.

Note that this PR was discussed in an IRC meeting here: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2019/bitcoin-core-dev.2019-01-10-19.00.log.html#l-175

Several contributors there were in favour of this approach because it's less code (@jonasschnelli , @jimpo , @jamesob , @laanwj ). As far as I'm concerned that means this approach still has far more concept ACKs than NACKs (but any of those contributors should speak up if they've changed their minds).

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 10, 2019

Note this PR didn't exist yet at the time of the IRC discussion. In any case, I tend to be more interested in reasoning behind NACKs, and how specific concerns can be addressed, than just their gut level responses. For example, Luke seemed to be concerned with comments and comment preservation in the dynamic configuration file, which could be addressed with code to read/write the dynamic configuration in ini format instead of json format. AJ seemed skeptical of dynamic configuration for bitcoind in general, and I definitely think there should be a way to turn off dynamic configuration (gray out dynamic configuration in gui, return errors from RPCs that would try to save a persistent setting).

@luke-jr

This comment has been minimized.

Copy link
Member

luke-jr commented Jul 10, 2019

Several contributors there were in favour of this approach because it's less code

But it's not really. UniValue is more code than updating the INI file. And until now, we have had a policy (that IMO we should keep) to only use UniValue in the JSON-RPC code.

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 10, 2019

we have had a policy (that IMO we should keep) to only use UniValue in the JSON-RPC code

@luke-jr, I think I've heard of this policy before, but maybe didn't take it seriously. Does it come from an objection to the JSON format, or problems with the univalue implementation, or something else? I wouldn't want univalue to used in application code (in wallet code or consensus code) because string lookups and variant typing lead to c++ code that's verbose, messy, and fragile. But at an API layer, or for accessing config data, univalue seems pretty well-suited, so I wonder what the concerns about it are or were.

@practicalswift

This comment has been minimized.

Copy link
Member

practicalswift commented Jul 10, 2019

Concept ACK

Rationale:

Standard format. The bitcoin.conf format works pretty well as a manual configuration mechanism. But the format is unspecified and unstable, and less appealing for use as persistent key/value store compared to JSON which is well understood and supported by many tools and libraries.

@NicolasDorier

This comment has been minimized.

Copy link
Member

NicolasDorier commented Jul 26, 2019

I think bitcoin.conf is easier to use than a json file by scripting languages.
It is easy to add one line to bitcoin.conf, but adding one node to a json file? It requires additional dependencies.

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 26, 2019

@NicolasDorier, can you clarify your use case? settings.json does not replace bitcoin.conf and should not be used for static configuration. It replaces QSettings (see #15936) and is used for dynamic configuration. It's database of settings updated dynamically while bitcoin is running, similar to peers.dat or fee_estimates.dat, only it happens to be in a human readable format.

@NicolasDorier

This comment has been minimized.

Copy link
Member

NicolasDorier commented Jul 26, 2019

@ryanofsky ah ok then this make sense. I was worried that some configuration thing start becoming spread in different files in different format, but this is not the case here.

@luke-jr

This comment has been minimized.

Copy link
Member

luke-jr commented Jul 26, 2019

Software should be modifying bitcoin_rw.conf/settings.json, not bitcoin.conf

@ryanofsky

This comment has been minimized.

Copy link
Contributor Author

ryanofsky commented Jul 26, 2019

Software should be modifying bitcoin_rw.conf/settings.json, not bitcoin.conf

It makes sense for deployment software to generate and modify bitcoin.conf files. Specifically configuration software like ansible, package managers like nix or guix, build scripts like docker files, should all be setting up their deployments through bitcoin.conf. Managing static software configurations is a big part of what these tools do and why they exist.

For bitcoin_rw.conf/settings.json, there are some exceptions I could think of, but mainly I think the only tool that will want to write these files is bitcoin itself. Partly this is due to safety, because with features like #15936 and #15937, settings can change any time while bitcoin is running, so external programs trying to modify these files could see their updates lost. But more importantly, a major motivation for wanting to add dynamic settings is to provide new, less cumbersome ways of updating settings than the raw file system. A user is less likely to screw up something like pruning when the way to enable it is to type a command like bitcoin-cli config prune 3gb, and get instant error checking and feedback, than to have to locate their datadir, open a text editor, make changes without running afoul of ini sections syntax and precedence rules, stop bitcoin, restart bitcoin, and check the logs to verify their changes picked up.

Bitcoin isn't pure server software that only needs static configuration, and it isn't pure application software that only needs dynamic configuration. It's P2P software that should support both static and dynamic configuration for different use cases. My expectation is that server admins and advanced users will use bitcoin.conf just like now and always, and for a lot of them to disable dynamic configuration. I would expect more typical users to have no bitcoin.conf file, or only a minimal one from their distro or package manager, and to manage settings through GUI and command interfaces without editing text files.

@laanwj laanwj added the Feature label Sep 30, 2019
laanwj added a commit that referenced this pull request Nov 8, 2019
083c954 Add settings_tests (Russell Yanofsky)
7f40528 Deduplicate settings merge code (Russell Yanofsky)
9dcb952 Add util::Settings struct and helper functions. (Russell Yanofsky)
e2e37cf Remove includeconf nested scope (Russell Yanofsky)
5a84aa8 Rename includeconf variables for clarity (Russell Yanofsky)
dc8e1e7 Clarify emptyIncludeConf logic (Russell Yanofsky)

Pull request description:

  This is a refactoring-only change that makes it easier to add a new settings source.

  This PR doesn't change behavior. The [`util_ArgsMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L626-L822) and [`util_ChainMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L843-L924) tests added in #15869 and #15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change.

  This change:

  - Makes it easier to add new settings sources that can get merged with existing sources (see 70675c3 from #15935).
  - Separates parsing of settings from merging of settings, and deduplicates merging code so it doesn't happen five different places ([GetArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L221-L244), [GetNetBoolArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L255-L261), [GetArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L460-L467), [IsArgNegated](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L482-L491), [GetUnsuitableSectionOnlyArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L343-L352)) in inconsistent ways.
  - Documents and tests current strange merging behaviors, so they be cleaned up in the future if resulting code simplifications and UX improvements warrant loss of backwards compatibility. The newly documented behaviors are: command line [ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/system.cpp#L323-L326) and [more ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L67-L72), and config file [reverse precedence](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L61-L65), [inconsistently applied top-level settings](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L55-L59), and [zombie values](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L101-L108).

  The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent `-wallet` setting are pretty straightforward and show how the new code can be extended:

  * 70675c3 from #15935 – _Add \<datadir>/settings.json persistent settings storage_
  * 04c80c4 from #15937 – _Add loadwallet and createwallet RPC load_on_startup options_

ACKs for top commit:
  ariard:
    ACK 083c954
  jnewbery:
    ACK 083c954
  jamesob:
    ACK 083c954

Tree-SHA512: 5d106746a44d64d3963c4ef3f4a2fa668a4bedcc9018d3ea12c86beae2fda48a0b036241665837f68685712366f70f2e1faba84d193fa1f456013503097b7659
sidhujag added a commit to syscoin/syscoin that referenced this pull request Nov 9, 2019
083c954 Add settings_tests (Russell Yanofsky)
7f40528 Deduplicate settings merge code (Russell Yanofsky)
9dcb952 Add util::Settings struct and helper functions. (Russell Yanofsky)
e2e37cf Remove includeconf nested scope (Russell Yanofsky)
5a84aa8 Rename includeconf variables for clarity (Russell Yanofsky)
dc8e1e7 Clarify emptyIncludeConf logic (Russell Yanofsky)

Pull request description:

  This is a refactoring-only change that makes it easier to add a new settings source.

  This PR doesn't change behavior. The [`util_ArgsMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L626-L822) and [`util_ChainMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L843-L924) tests added in bitcoin#15869 and bitcoin#15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change.

  This change:

  - Makes it easier to add new settings sources that can get merged with existing sources (see 70675c3 from bitcoin#15935).
  - Separates parsing of settings from merging of settings, and deduplicates merging code so it doesn't happen five different places ([GetArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L221-L244), [GetNetBoolArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L255-L261), [GetArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L460-L467), [IsArgNegated](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L482-L491), [GetUnsuitableSectionOnlyArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L343-L352)) in inconsistent ways.
  - Documents and tests current strange merging behaviors, so they be cleaned up in the future if resulting code simplifications and UX improvements warrant loss of backwards compatibility. The newly documented behaviors are: command line [ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/system.cpp#L323-L326) and [more ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L67-L72), and config file [reverse precedence](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L61-L65), [inconsistently applied top-level settings](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L55-L59), and [zombie values](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L101-L108).

  The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent `-wallet` setting are pretty straightforward and show how the new code can be extended:

  * 70675c3 from bitcoin#15935 – _Add \<datadir>/settings.json persistent settings storage_
  * 04c80c4 from bitcoin#15937 – _Add loadwallet and createwallet RPC load_on_startup options_

ACKs for top commit:
  ariard:
    ACK 083c954
  jnewbery:
    ACK 083c954
  jamesob:
    ACK 083c954

Tree-SHA512: 5d106746a44d64d3963c4ef3f4a2fa668a4bedcc9018d3ea12c86beae2fda48a0b036241665837f68685712366f70f2e1faba84d193fa1f456013503097b7659
@jnewbery

This comment has been minimized.

Copy link
Member

jnewbery commented Nov 11, 2019

The base PR #15934 is merged. Looking forward to reviewing this once it's rebased!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.