Export/import of extras:

[yakutovicha]

fixes #1761

1. Extras are now exported together with other data
2. Extras are now fully imported if the corresponding node did not exist
3. In case the imported node already exists in a database the following
   logic may be chosen to import the extras:
   • keep_existing (default): keep extras with different names (old and imported
     ones), keep old value in case of name collision
   • update_existing: -/-/-, keep new value in case of name collision
   • ask: -/-/-, ask what to do in case of the name collision
   • mirror: completely overwrite the extras by the imported ones
   • none: keep old extras

[coveralls]

Coverage increased (+0.03%) to 69.738% when pulling 1571b3d on yakutovicha:provenance_redesign into b85ac3e on aiidateam:provenance_redesign.

[sphuber]

I proposed some changes. As an additional comment, if I understand correctly, the extras import mode only affects nodes that are already present in the database. Specifically the none option will not import any of the extras in the archive. But shouldn't there also be a general option to not import any extras, regardless of whether the node is already present in the database or not?

[yakutovicha]

@sphuber

I wanted to discuss with you this "general option" to import extras. I see two ways:

1. If --extras-import-mode is set to 'none' it will not import any extras (independently if the node is present already or not)
2. Add another option, say --import-new-extras that will only affect the extras of a new nodes

But shouldn't there also be a general option to not import any extras, regardless of whether the node is already present in the database or not

I would prefer to not define one option that will affect both existing and new nodes, because in this way you can not, for example, import new extras for the existing nodes and do not import extras for the new nodes.

Of course, people may never need that, but I think if to add a new option it should allow to any combination of behavior with regard to the new and existing nodes.

[yakutovicha]

Apart from that last issue, I've already implemented all your suggestions

[sphuber]

I agree that we probably want to have a solution that allows the user to choose all possible options, so that would exclude the first one. I would be fine with adding an additional option that exclusively effects the import behavior of extras for new nodes. @giovannipizzi what do you think?

[yakutovicha]

Also, for the moment all extras are exported. Let me know I should add yes/no option for the extras export.

[sphuber]

I spoke to @giovannipizzi and he agrees that we should have all combinations available to (power) users. So we should have two flags, --extras-mode-new and --extras-mode-existing with sensible defaults. Since having to deal with two flags can be quite cumbersome, we should have a "compound" flag --extras-mode, that sets the values of these flags. This should be possible in click. Maybe for now this is too much work, but when we close this PR we should open an issue for v1.1.0.

So if you can change the name of the current flag and implement the second --extras-mode-new we should be good to go. Thanks!

[yakutovicha]

@sphuber all python 2 tests failed because of the same error:

Warning, treated as error:
autodoc: failed to import module u'test_postgres' from module u'aiida.backends.tests.manage.external'; the following exception was raised:
cannot import name timezone
make: *** [default] Error 2

and this has nothing to do with my changes

[sphuber]

Yeah there seems to be a problem with pgtest which calls somewhere from datetime import timezone as Timezone which started failing for some reason. There are other PR's with the same problem. I will take a look to see if I can figure out what is going on

[sphuber]

Found the problem: the package pg8000 which is a dependency of pgtest just made a new release v1.13.0 in which they dropped python 2 support. This means their line from datetime import timezone which is only supported in py3 fails for our python 2 setup. The problem is really that they used a minor version release to drop support of py2 🤦‍♂️ I will make an issue for this and fix it. We will have to wait for @szoupanos PR to be merged anyway. So for now there is nothing you have to do. Unless you want to tests your PR then you can manually insert in the requirements pg8000<=1.12.4 and run again.

[sphuber]

@yakutovicha I opened a PR to fix the problem, so once it is merged you can rebase so your tests will run again

[yakutovicha]

thanks @sphuber, just merged your changes

[yakutovicha]

I made the requested changes.

@sphuber and @szoupanos let me know if something else still needs to be done

[sphuber]

Thanks a lot @yakutovicha great work