Programmatically update `config.yaml`

[ddundo]

It looks like there is currently no way to update config.yaml that isn't manual; i.e., we have to open config.yaml with a text editor and update it that way. This can easily lead to errors (as it did in #58) and prevents automating tasks (e.g., I cannot add advanced_test_tw.ipynb to tests in #63).

I think we should:

• Add a pygem.setup.config.update_config function to do this
• Add checks that keys in config.yaml are correct (i.e., that there is not a typo) and that their values have expected type.

And while addressing this I think the whole config module should be rewritten in an object-oriented way.

[ddundo]

@btobers said that he will add a necessary flag to run_calibration.py so that advanced_test_tw.ipynb can be run automatically.

[btobers]

@ddundo I missed this issue before. I like the idea of creating a pygem.setup.config.update_config function which could update keys and would thus allow the advanced_test_tw.ipynb notebook to be added to the tests. I suppose either option - this or creating another flag/argument in run_calibration.py would work.

[ddundo]

Since config.yaml has so many options (over 250 it seems), I think it makes sense to have a single function like update_config that could update any one of those. And we could still keep the option to set them from the command line. So we could do something like this:

import argparse
import yaml

def update_config(config_path, updates):
    # Read the existing config file
    with open(config_path, 'r') as file:
        config = yaml.safe_load(file)

    # Update the config with the new values
    for key, value in updates.items():
        keys = key.split('.')
        d = config
        for k in keys[:-1]:
            d = d[k]
        d[keys[-1]] = value

    # Write the updated config to a new file
    with open('newconfig.yaml', 'w') as file:
        yaml.safe_dump(config, file)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--config', default='config.yaml', help='Path to the config file')
    parser.add_argument('updates', nargs='*', help='Key-value pairs to update in the config file')

    args = parser.parse_args()

    # Parse the updates into a dictionary
    updates = {}
    for update in args.updates:
        key, value = update.split('=')
        updates[key] = value

    update_config(args.config, updates)

if __name__ == '__main__':
    main()

If you save this to a file called parser_test.py and then run, for example, python3 parser_test.py --config PyGEM/config.yaml climate.ref_startyear=2015 oggm.has_internet=true you will see that this creates a new file newconfig.yaml with the updated ref_startyear and has_internet options in the correct spots. This is just quick and dirty, but you get the point :)

[btobers]

That makes sense. I like the idea of making an update_config function, and having a CLI update option.

I was in the process of implementing this and making it more object oriented. Let me know what you think of this below:

import os
import shutil
import yaml
import argparse
from ruamel.yaml import YAML

class ConfigManager:
    def __init__(self, config_filename='config.yaml', base_dir=None):
        self.config_filename = config_filename
        self.base_dir = base_dir or os.path.join(os.path.expanduser('~'), 'PyGEM')
        self.config_path = os.path.join(self.base_dir, self.config_filename)
        self.package_dir = os.path.dirname(__file__)
        self.source_config_path = os.path.join(self.package_dir, self.config_filename)
    
    def ensure_config(self, overwrite=False):
        if os.path.isfile(self.config_path) and not overwrite:
            return
        
        if os.path.isfile(self.config_path) and overwrite:
            overwrite = self._prompt_overwrite()
        
        if not os.path.isfile(self.config_path) or overwrite:
            os.makedirs(self.base_dir, exist_ok=True)
            shutil.copy(self.source_config_path, self.config_path)
            print(f"Copied default configuration to {self.config_path}")
    
    def read_config(self):
        """Read the configuration file and return its contents as a dictionary."""
        with open(self.config_path, 'r') as f:
            return yaml.safe_load(f)
    
    def update_config(self, updates):
        """Update multiple keys in the YAML configuration file while preserving quotes."""
        yaml = YAML()
        yaml.preserve_quotes = True  # Preserve quotes around string values
    
        try:
            with open(self.config_path, 'r') as file:
                config = yaml.load(file)
            
            # Update the config with the new values
            for key, value in updates.items():
                keys = key.split('.')
                d = config
                for k in keys[:-1]:
                    d = d.setdefault(k, {})  # Ensure intermediate keys exist
                d[keys[-1]] = value
                
            with open(self.config_path, 'w') as file:
                yaml.dump(config, file)
        except Exception as e:
            print(f"Error updating config: {e}")
    
    def _prompt_overwrite(self):
        """Prompt the user for confirmation before overwriting the config file."""
        while True:
            response = input(f"Configuration file already exists ({self.config_path}). Overwrite? (y/n): ").strip().lower()
            if response in ['y', 'yes']:
                return True
            elif response in ['n', 'no']:
                return False
            print("Invalid input. Please enter 'y' or 'n'.")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--config', default='config.yaml', help='Path to the config file')
    parser.add_argument('updates', nargs='*', help='Key-value pairs to update in the config file')
    
    args = parser.parse_args()
    
    # Parse the updates into a dictionary
    updates = {}
    for update in args.updates:
        key, value = update.split('=')
        updates[key] = value
    
    config_manager = ConfigManager(config_filename=args.config)
    config_manager.update_config(updates)

if __name__ == '__main__':
    main()

[ddundo]

Very cool that this preserves comments! And that looks like a good start. At the moment this updates options with a string though, i.e., ref_startyear: '2015' instead of ref_startyear: 2015. But yeah, please open a PR when you want me to take a look (even if it's unfinished) :)

[btobers]

Fixed in PR #78

[btobers]

Closed in #78.