Add Parameters to Benchmarks #103
Conversation
Add the Parameters to a 'BenchmarkRecord' by appending it to the results. Omit the type information from the `Variable` object and only retain variable name and value.
src/nnbench/runner.py
Outdated
| @@ -263,6 +264,7 @@ def run( | |||
| "date": datetime.now().isoformat(timespec="seconds"), | |||
| "error_occurred": False, | |||
| "error_message": "", | |||
| "parameters": {**bmdefaults, **bmparams}, | |||
There was a problem hiding this comment.
Does this ensure that the defaults are overwritten by the bmparams?
I think something like bmdefaults.update(bmparams) would be easier to read.
There was a problem hiding this comment.
Yes, the entires are unpacked in order with the latter overwriting the former in case of colliding keys.
dict.update does not return the merged dict, but just updates inplace.
We couid do dict(bmdefaults, **bmparams) , though I don't think that is easier to read.
Alternatively, do an update on bmdefaults and then pass only that. That is an extra line though and we have a bmdefaults flying around that does not contain the defaults.
There was a problem hiding this comment.
I think this does what it needs to do: For any variable, if the default is used, it's not in bmparams, so the update does nothing - and for every overridden default, the update() overwrites with the argument given in the params dict.
So the order should be "use defaults, then update with params", which is what I suggested (you can also write bmdefaults | bmparams).
There was a problem hiding this comment.
If you need a mental model, it helps to think about what happens once the params are passed into the function - its default keyword arguments are already there, and everything else gets overridden by the incoming parameters. So it is indeed bmdefault.update(bmparams) we're looking for here (you can even use that as the dict value, since we do not explicitly use it again afterwards).
Co-authored-by: Nicholas Junge <n.junge@appliedai.de>
What's left is to decide what to do with parameter types that aren't as easily ingestible into a sink as builtin datatypes.
Closes #101
Add the Parameters to a 'BenchmarkRecord' by appending it to the results. Omit the type information from the
Variableobject and only retain variable name and value.