Separate automated schema loading from config options #9307
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jdavcs
added
status/WIP
kind/refactoring
Rationale: group related items together. Constants OPTION_DEFAULTS and UNKNOWN_OPTION are closely related and are used in the same context. OPTION_DEFAULTS is only used in `schema.py` and `config_manage.py`; the latter imports items from `schema.py`. Thus, it is reasonable to place both constants in the same module.
This commit drops the exception that guarded against accidental overwriting of existing attributes, and instead ignores the setattr call, logging a debug message (which was initialy proposed in galaxyproject#8728). Rationale: with the previous version, base configuration properties (config_dir, data_dir, etc.) had to be set *after* this method was called (otherwise an exeption was raised), but *before* any path resolving code (because these properties are used for path resolution). As a result, config loading required a rigid sequence of steps: 1: load defaults and kwargs, create all config attributes. 2: set base configuration properties [CANNOT MOVE THIS] 3: resolve paths, do the rest: load individual properties, etc. However, setting base configuration properties is a special case that is *independent of schema loading*, is handled separately (in set_config_base), and takes presendence over schema loading procedures. This modification makes config loading more flexible, and logical: 1. set base configuration properties 2. do the rest (now any refactoring can go here)
Main configuration module is doing too much. Separate automated schema loading from loading individual config options and misc. special cases.
Also, _raw_config is no longer "private", so remove underscore
|
Thank you for reviewing, @bgruening! |
nsoranzo
reviewed
Feb 10, 2020
nsoranzo
reviewed
Feb 11, 2020
nsoranzo
reviewed
Feb 11, 2020
Co-Authored-By: Nicola Soranzo <nicola.soranzo@gmail.com>
As per code review by @nsoranzo
nsoranzo
reviewed
Feb 11, 2020
| resolve(key) | ||
| def _load_config(self, kwargs): | ||
| loader = ConfigurationLoader() | ||
| loader.load(self, kwargs) |
There was a problem hiding this comment.
Here loader.load() has the side effect of modifying values in self, which makes me wonder if ConfigurationLoader is the right abstraction here.
I don't have an alternative solution though.
There was a problem hiding this comment.
That side effect is by design: creating schema-loaded attributes on the config object is one of the loader's primary responsibilities. You have a point, of course: by doing so the loader manipulates the internal structure of the config object. Of course, these two end up being very tightly coupled.
I'm not sure whether this is a good enough solution; it might be. Or maybe, alternatively, the loader could prepare a dict and supply it to the config, and then the config would set its own attributes based on that data (and that data would be the current raw_config). That might be a better approach. I'll give this some more thought tomorrow.
There was a problem hiding this comment.
If they are so tightly coupled there is no point in separating some methods in a different class, IMO. This kind of side effects need to be very well documented, and will still surprise other developers.
The alternative approach could work, I guess. Maybe @jmchilton has suggestions? I tend to refer to him as expert in class/module separation 🙇♂️
There was a problem hiding this comment.
@nsoranzo you are right - it is not the right abstraction. We can do better. I'll close this in favor of a follow-up. Thank you for the detailed review!
|
I'm closing this in favor of #9371 (currently WIP) |
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Feb 15, 2020
Rationale: group related items together. Constants OPTION_DEFAULTS and UNKNOWN_OPTION are closely related and are used in the same context. OPTION_DEFAULTS is only used in `schema.py` and `config_manage.py`; the latter imports items from `schema.py`. Thus, it is reasonable to place both constants in the same module. (cherry-picked from closed galaxyproject#9307)
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Feb 15, 2020
This commit drops the exception that guarded against accidental overwriting of existing attributes, and instead ignores the setattr call, logging a debug message (which was initialy proposed in galaxyproject#8728). Rationale: with the previous version, base configuration properties (config_dir, data_dir, etc.) had to be set *after* this method was called (otherwise an exeption was raised), but *before* any path resolving code (because these properties are used for path resolution). As a result, config loading required a rigid sequence of steps: 1: load defaults and kwargs, create all config attributes. 2: set base configuration properties [CANNOT MOVE THIS] 3: resolve paths, do the rest: load individual properties, etc. However, setting base configuration properties is a special case that is *independent of schema loading*, is handled separately (in set_config_base), and takes precendence over schema loading procedures. This modification makes config loading more flexible, and logical: 1. set base configuration properties 2. do the rest (now any refactoring can go here) (cherry-picked from closed galaxyproject#9307)
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Mar 2, 2020
Rationale: group related items together. Constants OPTION_DEFAULTS and UNKNOWN_OPTION are closely related and are used in the same context. OPTION_DEFAULTS is only used in `schema.py` and `config_manage.py`; the latter imports items from `schema.py`. Thus, it is reasonable to place both constants in the same module. (cherry-picked from closed galaxyproject#9307)
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Mar 2, 2020
This commit drops the exception that guarded against accidental overwriting of existing attributes, and instead ignores the setattr call, logging a debug message (which was initialy proposed in galaxyproject#8728). Rationale: with the previous version, base configuration properties (config_dir, data_dir, etc.) had to be set *after* this method was called (otherwise an exeption was raised), but *before* any path resolving code (because these properties are used for path resolution). As a result, config loading required a rigid sequence of steps: 1: load defaults and kwargs, create all config attributes. 2: set base configuration properties [CANNOT MOVE THIS] 3: resolve paths, do the rest: load individual properties, etc. However, setting base configuration properties is a special case that is *independent of schema loading*, is handled separately (in set_config_base), and takes precendence over schema loading procedures. This modification makes config loading more flexible, and logical: 1. set base configuration properties 2. do the rest (now any refactoring can go here) (cherry-picked from closed galaxyproject#9307)
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Mar 9, 2020
Rationale: group related items together. Constants OPTION_DEFAULTS and UNKNOWN_OPTION are closely related and are used in the same context. OPTION_DEFAULTS is only used in `schema.py` and `config_manage.py`; the latter imports items from `schema.py`. Thus, it is reasonable to place both constants in the same module. (cherry-picked from closed galaxyproject#9307)
jdavcs
added a commit
to jdavcs/galaxy
that referenced
this pull request
Mar 9, 2020
This commit drops the exception that guarded against accidental overwriting of existing attributes, and instead ignores the setattr call, logging a debug message (which was initialy proposed in galaxyproject#8728). Rationale: with the previous version, base configuration properties (config_dir, data_dir, etc.) had to be set *after* this method was called (otherwise an exeption was raised), but *before* any path resolving code (because these properties are used for path resolution). As a result, config loading required a rigid sequence of steps: 1: load defaults and kwargs, create all config attributes. 2: set base configuration properties [CANNOT MOVE THIS] 3: resolve paths, do the rest: load individual properties, etc. However, setting base configuration properties is a special case that is *independent of schema loading*, is handled separately (in set_config_base), and takes precendence over schema loading procedures. This modification makes config loading more flexible, and logical: 1. set base configuration properties 2. do the rest (now any refactoring can go here) (cherry-picked from closed galaxyproject#9307)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update: this is ready for review.
Delegate loading+processing configuration from schema to separate module.
config/__init__.pyis doing too much. This is an attempt to separate 2 different responsibilities: (1) automatically loading and processing all config properties from the schema; and (2) setting individual config properties and handling special cases. The former is delegated to aConfigurationLoadermodule. The idea is to use this loader module for any further automated schema processing, whereasGalaxyAppConfigurationis to be used for processing individual config properties (as it did before).Last year's modifications to the config system, while simplifying configuration loading, have done the opposite to the configuration module's code. This modification is an attempt to address this problem.
(Edit: more detailed rationale in commit messages)