Global parameter #206

benjello · 2016-01-13T15:01:34Z

For now I am able to introduce a DEBUG parameters.
@gdementen could you tell me if this seems ok and if I should proceed to the inclusion of any parameters set to a non iterable variable (or a variable that have with neither path nor defined in the node globals of the hdf file).

gdementen · 2016-01-29T15:59:48Z

liam2/entities.py

@@ -40,7 +40,12 @@ def global_symbols(globals_def):
    # entity. I guess they should have a class of their own
    symbols = {}
    for name, global_def in globals_def.iteritems():
-        global_type = global_def.get('fields')
+        try:


I'd prefer an isinstance check in this case

gdementen · 2016-01-29T16:20:31Z

Ok, so this PR looks good in principle. The implementation is a bit ugly to my taste (no real objective criticism), but I can fix that myself, but that validator thing is a no-go. Might need a separate PR for that. I would personally write that as a class:

class Or(object):
    def __init__(self, choices):
        ....

# and in the layout, you would use:
yaml_layout = {
         'import': None,
         'globals': {
          '*': Or(str, int, float, bool, {
                 'path': str,
                 'type': str,
                 'fields': [{
                     '*': None  # Or(str, {'type': str, 'initialdata': bool})
                 }],
                 'oldnames': {
                     '*': str
                 },
                 'newnames': {
                     '*': str
                 },
                 'invert': [str],
                 'transposed': bool
             })

or something like that...

benjello · 2017-04-19T14:41:43Z

@gdementen : I made some changes following your remarks. Feel free to ask more changes.

gdementen · 2017-04-19T15:09:31Z

liam2/data.py

        globals_node = getattr(input_root, 'globals', None)
        for name, global_def in globals_def.iteritems():
            # already loaded from another source (path)
            if name in globals_data:
                continue

            if name not in globals_node:
+                if isinstance(global_def, numbers.Number) or isinstance(global_def, bool):


wouldn't if np.isscalar(global_def) be more robust? I would move the check above, so that None of the code below is executed (it seems useless in this case). In fact, I would just add these lines just below globals_data = load_path_globals(globals_def):

globals_data.update({k: v for k, v in globals_def.iteritems() if np.isscalar(v)})

EDIT: In fact, I think this code is completely useless now that the constants are handled before the loop (currently in load_path_globals, but if you handle them in handle_constant_globals, that would be the same.

gdementen · 2017-04-19T15:52:15Z

liam2/simulation.py

+                    # provided in the file)
+                    v["fields"] = fields_yaml_to_type(v["fields"])
+                globals_def[k] = v
+            except TypeError:


I would rather use an np.isscalar(v) check than a try except

gdementen · 2017-04-19T15:56:06Z

liam2/data.py

@@ -710,12 +711,15 @@ def load_path_globals(globals_def):
    localdir = config.input_directory
    globals_data = {}
    for name, global_def in globals_def.iteritems():
-        if 'path' not in global_def:
+        if 'path' not in global_def and 'value' not in global_def:


"constant" globals shouldn't be handled in load_path_globals. Either in index_tables directly or in a separate method "handle_constant_globals".

gdementen · 2017-04-19T16:00:36Z

liam2/data.py

@@ -853,6 +858,11 @@ def prepare(self, globals_def, entities, input_dataset, start_period):
                output_globals = output_file.create_group("/", "globals",
                                                          "Globals")
                for k, g_def in globals_def.iteritems():
+                    # Do not save global constants ie those who have a value
+                    # attribute
+                    if 'value' in g_def:


maybe change the test to if 'value' in g_def or 'path' in g_def: so that the if 'path' not in g_def test can be removed?

gdementen · 2017-04-19T16:01:58Z

liam2/importer.py

@@ -562,6 +562,9 @@ def load_def(localdir, ent_name, section_def, required_fields):
                        "type and fields sections are mutually exclusive"
                        % ent_name)

+    if "value" in section_def:
+        return 'scalar', section_def["value"]


remove this change

gdementen · 2017-04-19T16:04:23Z

liam2/utils.py

@@ -1081,6 +1082,8 @@ def validate_value(v, target, context):
        validate_dict(v, target, context)
    elif isinstance(target, list):
        validate_list(v, target, context)
+    elif isinstance(target, numbers.Number) or isinstance(target, bool):


this shouldn't be needed since you are using None as validator for the 'value' keyword

gdementen

Please also:

revert your changes to demo.h5 and small.h5 (unless you have a good reason, but then it needs to be documented)
add a changelog line
add the corresponding documentation

gdementen · 2017-09-22T11:37:38Z

liam2/data.py

@@ -853,8 +869,12 @@ def prepare(self, globals_def, entities, input_dataset, start_period):
                output_globals = output_file.create_group("/", "globals",
                                                          "Globals")
                for k, g_def in globals_def.iteritems():
-                    if 'path' not in g_def:
-                        anyarray_to_disk(output_globals, k, globals_data[k])
+                    # Do not save global constants ie those who have a value


Do not save global constants (ie those who have a value attribute) nor globals loaded from external (.csv) files.

gdementen · 2017-09-22T11:38:57Z

liam2/tests/functional/simulation.yml

@@ -2796,6 +2808,11 @@ entities:
                              fields=[('PERIOD', int), ('INTFIELD', int), ('FLOATFIELD', float)])
                - assertEqual(table.shape, (31,))

+            test_global_parameter:
+                - assertTrue(BOOL_CONSTANT)
+                - assertTrue(FLOAT_CONSTANT == 3.1415)


use assertEqual instead?

gdementen · 2017-09-22T11:50:07Z

liam2/data.py

+def handle_constant_globals(globals_def):
+    globals_data = {}
+    for name, global_def in globals_def.iteritems():
+        if 'value' not in global_def:


unsure restricting to scalars adds any value. Using constant array globals does not seem like such a bad idea.

the "or isinstance(value, bool)" part is useless (AFAICT) since np.isscalar(True) is True and np.isscalar(False) is True
So I would rewrite the content of the loop to something like:

if 'value' in global_def: globals_data[name] = global_def['value']

gdementen · 2017-09-22T11:54:34Z

liam2/entities.py

@@ -41,7 +41,12 @@ def global_symbols(globals_def):
    # entity. I guess they should have a class of their own
    symbols = {}
    for name, global_def in globals_def.iteritems():
-        global_type = global_def.get('fields')
+        if isinstance(global_def, dict):


this change is useless AFAICT: it would be useful if you could write:

BOOL_CONSTANT: True FLOAT_CONSTANT: 3.1415 INT_CONSTANT: 42

but the current "yaml validator" forbids that.

EDIT: I have just seen this is useful after all (because of the code in from_str) but I would rather have that changed. and Ideally, the case where value is given, type should be optional (ie inferred using type(value))

gdementen · 2017-09-22T12:00:37Z

liam2/simulation.py

@@ -241,6 +242,12 @@ def from_str(cls, yaml_str, simulation_dir='',
        #                'MIG': {'type': int}}
        globals_def = {}
        for k, v in content.get('globals', {}).iteritems():
+            if 'value' in v:
+                if np.isscalar(v['value']) or isinstance(v['value'], bool):


unsure restricting to scalars adds any value

isinstance(v['value'], bool) seems useless

gdementen · 2017-09-22T12:12:33Z

liam2/simulation.py

@@ -241,6 +242,12 @@ def from_str(cls, yaml_str, simulation_dir='',
        #                'MIG': {'type': int}}
        globals_def = {}
        for k, v in content.get('globals', {}).iteritems():
+            if 'value' in v:


this whole block of change could be removed IMO because it is mostly unneeded. All it does is transform {'INT_CONSTANT': {'value': 42, 'type': 'int'}} to {'INT_CONSTANT': 42}

I would rather avoid doing that and simply handle the "value" case in globals_symbols. Besides, I prefer if from_str is does not change the structure of the globals definition, just convert string (types) to real types.

benjello · 2017-09-22T16:48:58Z

Modifications and rebasing done.

gdementen · 2017-09-22T18:31:32Z

liam2/data.py

        kind, info = load_def(localdir, name, global_def, [])
        if kind == 'table':
            fields, numlines, datastream, csvfile = info
            array = stream_to_array(fields, datastream, numlines)
+        elif kind == 'scalar':


are these two lines of any use? I do not see anything producing that kind = "scalar"???

gdementen · 2017-09-22T18:33:56Z

doc/usersguide/source/changes/version_0_12.rst.inc

@@ -43,6 +43,21 @@
 * implemented experimental align(link=) to use proportions in combination with the Chenard algorithm. It needs more
  testing though.

+* implemented a way of declaring global constants. These constant should be explicitely typed. (partially closes :issue:`203`): ::


why partially?

We cannot use:

INT_CONSTANT: 42

I consider the feature implemented, even though the syntax is not ideal. We can still improve the syntax later, but I will close the issue anyway.

benjello · 2017-09-24T22:20:02Z

Rebase done

gdementen

LGTM. Thanks!

gdementen reviewed Jan 29, 2016
View reviewed changes

gdementen modified the milestone: 0.12 Apr 19, 2016

gdementen added the enhancement label Apr 19, 2016

benjello force-pushed the global_parameter branch from 6db15eb to 8452875 Compare April 19, 2017 09:01

gdementen reviewed Apr 19, 2017

View reviewed changes

gdementen reviewed Sep 22, 2017

View reviewed changes

Improve error message

9781618

benjello force-pushed the global_parameter branch 2 times, most recently from 3888b22 to e2f82b7 Compare September 24, 2017 22:16

Be able to set a global parameter in the yaml file

88542c0

benjello force-pushed the global_parameter branch from e2f82b7 to 88542c0 Compare September 24, 2017 22:25

gdementen approved these changes Sep 25, 2017

View reviewed changes

gdementen merged commit 08f6f0c into liam2:master Sep 25, 2017

benjello deleted the global_parameter branch September 25, 2017 10:17

benjello mentioned this pull request Sep 26, 2017

Set value of a global parameter in the yaml file should be possible #203

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Global parameter #206

Global parameter #206

benjello commented Jan 13, 2016

gdementen Jan 29, 2016

gdementen commented Jan 29, 2016

benjello commented Apr 19, 2017

gdementen Apr 19, 2017

gdementen Apr 19, 2017

gdementen Apr 19, 2017

gdementen Apr 19, 2017

gdementen Apr 19, 2017

gdementen Apr 19, 2017

gdementen left a comment

gdementen Sep 22, 2017

gdementen Sep 22, 2017

gdementen Sep 22, 2017

gdementen Sep 22, 2017

gdementen Sep 22, 2017

gdementen Sep 22, 2017

benjello commented Sep 22, 2017

gdementen Sep 22, 2017

gdementen Sep 22, 2017

benjello Sep 23, 2017

gdementen Sep 25, 2017

benjello commented Sep 24, 2017

gdementen left a comment

Global parameter #206

Global parameter #206

Conversation

benjello commented Jan 13, 2016

Choose a reason for hiding this comment

gdementen commented Jan 29, 2016

benjello commented Apr 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gdementen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjello commented Sep 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjello commented Sep 24, 2017

gdementen left a comment

Choose a reason for hiding this comment