-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement a PluginVersionProvider
for processes
#3131
Implement a PluginVersionProvider
for processes
#3131
Conversation
I tried to couple the This implementation only adds the version information to process nodes and therefore the best place is to have this information inserted in the
In the future the key N.B.: in this implementation, only Pinging @chrisjsewell @giovannipizzi @ltalirz @zhubonan for feedback about current proposal |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a small change required, thanks!
aiida/engine/processes/process.py
Outdated
@@ -633,6 +633,12 @@ def _setup_db_record(self): | |||
|
|||
def _setup_metadata(self): | |||
"""Store the metadata on the ProcessNode.""" | |||
try: | |||
version_info = self.runner.plugin_version_provider.get_version_info(self) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the current implementation I don't think the try/except is necessary
A comment: if we want to support this also for Data nodes (not a bad idea) we should probably have another column ( |
@chrisjsewell @ltalirz @zhubonan for me this (apart for a minor fix) is good to merge. Do you see anything we are missing in terms of important information to keep? Otherwise we'll merge mid next week. |
aiida/plugins/utils.py
Outdated
KEY_VERSION_ROOT = 'version' | ||
KEY_VERSION_CORE = 'core' # The version of `aiida-core` | ||
KEY_VERSION_PLUGIN = 'plugin' # The version of the plugin top level module, e.g. `aiida-quantumespresso` | ||
KEY_VERSION_ENTRY_POINT = 'entry_point' # The version of the specific corresponding entry point |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe remove this line since it's not yet used
Hey @sphuber, looks good ta :) Yeh My only other thought, which is probably overkill, is that it might be nice to also read/store an (optional) |
fcb7f98
to
1624fcc
Compare
I think it is a nice thing to have as at the moment no explicit information has been stored about the version. @chrisjsewell I think that the module level aiida-core/aiida/orm/nodes/node.py Lines 1091 to 1099 in 6d4f1ec
However, this does mean that if you bump up the module level I also agree with @chrisjsewell that it is quite likely that a |
Thanks a lot for the feedback guys. I see the use-case of being able to re-use The only thing that bothers me here is a simple problem of "language". I find it becomes difficult to talk about the different versions because things like "package", "module" and "plugin" are ambiguous and can refer to various things. I think it would be very valuable if we can define these now, such that extending the functionality in the future will be easier, when we know what we are talking about.
So how do we define the versions and call the following things? (using
The latter could be called |
I would vote not to include module-level versioning at this point.
I'm not sure we even need to plan ahead to include such versioning at this point. On comment do I have is that providing the version of two packages (core + package where the process is defined) is not only an incomplete approximation of the python package environment but even of the "aiida python package environment". E.g. consider I publish a workflow package I'm not suggesting to go beyond 2 packages now - but this is just to say that, in the medium term, we may not just want to have a
|
P.S. Perhaps more important than defining the naming scheme or the implementation would be to define the purpose of storing the version information.
This is quite vague - if we can agree what we would like to use the version information for (e.g. its relation to hashing), we also get a clearer picture of where this will go in the future. |
Regarding the name "package": the
This is fine though, because the
The amount of discussion here shows that just coming up with the definitions of the various versions and what they should mean is difficult. Any business logic that should then act on those versions will inherently be even more complicated and probably error prone. The goal here was then precisely not to tackle the whole problem, but rather provide a minimal subset, with the least constraints/complications for the greatest informational gain. This is what I meant with the statement:
The protocol for how these version attributes are determined (in this current proposal) is very easy to define (but therefore is very limited in terms of configurability/granularity). |
You're right - but I'm not aware of any aiida plugin that versions subfolders separately.
Indeed - any point chosen to stop here is arbitrary to some degree. You chose to stop after aiida-core and the version of the package that defines the process, and I agree that it makes the most sense for now (it's just not the only possible choice). |
d2acba4
to
ee505ab
Compare
This new utility class is used by the `Runner` class to keep a mapping of certain process classes, either `Process` sub classes or process functions onto a dictionary of "version" information. This dictionary now includes the version of `aiida-core` that is running as well as the version of the top level module of the plugin, if defined. If the latter cannot be determined for whatever reason, only the version of `aiida-core` is returned. This information is then retrieved by the `Process` class during the creation process. It will store this information in the attributes of the process node. Currently, no logic in `aiida-core` will act on this information. Its sole purpose is to give the user slightly more info on with what versions of core and the plugin a certain process node was generated. The attributes can also be used in querying to filter nodes for a sub set generated with a specific version of the plugin. Note that "plugin" here refers to the entire package, e.g the entire `aiida-quantumespresso` plugin. Each plugin can contain multiple entry points for the various entry point categories, that are sometimes individually also referred to as plugins. In the future, the version dictionary returned by the `PluginVersionProvider` may be enriched with a specific `entry_point` version, once that level of version granulariy will become supported. For the time being it is not included.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
ee505ab
to
5a0e2d2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks again ;-)
This new utility class is used by the `Runner` class to keep a mapping of certain process classes, either `Process` sub classes or process functions onto a dictionary of "version" information. This dictionary now includes the version of `aiida-core` that is running as well as the version of the top level module of the plugin, if defined. If the latter cannot be determined for whatever reason, only the version of `aiida-core` is returned. This information is then retrieved by the `Process` class during the creation process. It will store this information in the attributes of the process node. Currently, no logic in `aiida-core` will act on this information. Its sole purpose is to give the user slightly more info on with what versions of core and the plugin a certain process node was generated. The attributes can also be used in querying to filter nodes for a sub set generated with a specific version of the plugin. Note that "plugin" here refers to the entire package, e.g the entire `aiida-quantumespresso` plugin. Each plugin can contain multiple entry points for the various entry point categories, that are sometimes individually also referred to as plugins. In the future, the version dictionary returned by the `PluginVersionProvider` may be enriched with a specific `entry_point` version, once that level of version granulariy will become supported. For the time being it is not included.
This new utility class is used by the `Runner` class to keep a mapping of certain process classes, either `Process` sub classes or process functions onto a dictionary of "version" information. This dictionary now includes the version of `aiida-core` that is running as well as the version of the top level module of the plugin, if defined. If the latter cannot be determined for whatever reason, only the version of `aiida-core` is returned. This information is then retrieved by the `Process` class during the creation process. It will store this information in the attributes of the process node. Currently, no logic in `aiida-core` will act on this information. Its sole purpose is to give the user slightly more info on with what versions of core and the plugin a certain process node was generated. The attributes can also be used in querying to filter nodes for a sub set generated with a specific version of the plugin. Note that "plugin" here refers to the entire package, e.g the entire `aiida-quantumespresso` plugin. Each plugin can contain multiple entry points for the various entry point categories, that are sometimes individually also referred to as plugins. In the future, the version dictionary returned by the `PluginVersionProvider` may be enriched with a specific `entry_point` version, once that level of version granulariy will become supported. For the time being it is not included.
Fixes #2664 and fixes #2988
This new utility class is used by the
Runner
class to keep a mappingof certain process classes, either
Process
sub classes or processfunctions onto a dictionary of "version" information. This dictionary
now includes the version of
aiida-core
that is running as well as theversion of the top level module of the plugin, if defined. If the latter
cannot be determined for whatever reason, only the version of
aiida-core
is returned.This information is then retrieved by the
Process
class during thecreation process. It will store this information in the attributes of
the process node. Currently, no logic in
aiida-core
will act on thisinformation. Its sole purpose is to give the user slightly more info on
with what versions of core and the plugin a certain process node was
generated. The attributes can also be used in querying to filter nodes
for a sub set generated with a specific version of the plugin.
Note that "plugin" here refers to the entire package, e.g the entire
aiida-quantumespresso
plugin. Each plugin can contain multiple entrypoints for the various entry point categories, that are sometimes
individually also referred to as plugins. In the future, the version
dictionary returned by the
PluginVersionProvider
may be enriched witha specific
entry_point
version, once that level of version granulariywill become supported. For the time being it is not included.