verdi import from URL broken

[ltalirz]

This is the current develop branch running under python 3.7

$ verdi import https://archive.materialscloud.org/file/2018.0001/v3/SSSP_efficiency_pseudos.aiida
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/types.py", line 530, in convert
    st = os.stat(rv)
FileNotFoundError: [Errno 2] No such file or directory: 'https://archive.materialscloud.org/file/2018.0001/v3/SSSP_efficiency_pseudos.aiida'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/leopold/Personal/Postdoc-MARVEL/repos/aiida/aiida_rmq/aiida/cmdline/params/types/path.py", line 75, in convert
    return super().convert(value, param, ctx)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/types.py", line 537, in convert
    ), param, ctx)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/types.py", line 69, in fail
    raise BadParameter(message, ctx=ctx, param=param)
click.exceptions.BadParameter: Path "https://archive.materialscloud.org/file/2018.0001/v3/SSSP_efficiency_pseudos.aiida" does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/leopold/Personal/Postdoc-MARVEL/repos/aiida/aiida_rmq/aiida/cmdline/params/types/path.py", line 87, in checks_url
    urllib.request.urlopen(url, data=None, timeout=self.timeout_seconds)
AttributeError: module 'urllib' has no attribute 'request'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/bin/verdi", line 10, in <module>
    sys.exit(verdi())
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1135, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 641, in make_context
    self.parse_args(ctx, args)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 940, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1469, in handle_parse_result
    value = self.full_process_value(ctx, value)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1438, in full_process_value
    value = self.process_value(ctx, value)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1428, in process_value
    return self.type_cast_value(ctx, value)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1417, in type_cast_value
    return _convert(value, (self.nargs != 1) + bool(self.multiple))
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1416, in _convert
    return tuple(_convert(x, level - 1) for x in value or ())
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1416, in <genexpr>
    return tuple(_convert(x, level - 1) for x in value or ())
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/core.py", line 1415, in _convert
    return self.type(value, self, ctx)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/site-packages/click/types.py", line 39, in __call__
    return self.convert(value, param, ctx)
  File "/Users/leopold/Personal/Postdoc-MARVEL/repos/aiida/aiida_rmq/aiida/cmdline/params/types/path.py", line 78, in convert
    return self.checks_url(value, param, ctx)
  File "/Users/leopold/Personal/Postdoc-MARVEL/repos/aiida/aiida_rmq/aiida/cmdline/params/types/path.py", line 88, in checks_url
    except (urllib.error.URLError, urllib.error.HTTPError, timeout):
AttributeError: module 'urllib' has no attribute 'error'

[CasperWA]

This is an error of your environment. I have seen this before (and I believe, so have you in a previous issue). if you create a new virtualenv and/or install aiida-core from a different path this should not occur - as demonstrated by the (passing) CI tests.

[ltalirz]

We noticed this issue on another AiiDA installation on @sponce 's machine, i.e. perhaps the reason for closing #3610 was not quite the actual reason...

This also still happens on his machine

$ python
Python 3.7.1 (default, Oct 23 2018, 14:07:42)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> urllib.__file__
'/Users/leopold/Applications/miniconda3/envs/aiida_rmq_py3/lib/python3.7/urllib/__init__.py'
>>> urllib.error.URLError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'urllib' has no attribute 'error'
>>> from urllib import error

Confirmed on python 3.6.9 and python 3.7.1

[ltalirz]

The issue is we are not importing urllib correctly, see also here

The remaining open question is why this test passes...

aiida-core/tests/cmdline/commands/test_import.py

Lines 175 to 188 in 6ce6cd4

	def test_import_old_url_archives(self):
	""" Test import of old URL archives
	Expected behavior: Automatically migrate to newest version and import correctly.
	"""
	archive = 'export_v0.3_no_UPF.aiida'
	version = '0.3'
	options = [self.url_path + archive]
	result = self.cli_runner.invoke(cmd_import.cmd_import, options)
	self.assertIsNone(result.exception, msg=result.output)
	self.assertEqual(result.exit_code, 0, msg=result.output)
	self.assertIn(version, result.output, msg=result.exception)
	self.assertIn('Success: imported archive {}'.format(options[0]), result.output, msg=result.exception)

I can confirm that the tests do not pass in my virtual environment when I run

pytest tests/cmdline/commands/test_import.py

.

[ltalirz]

Since this is related to how urllib is imported, the explanation may be that other parts of the test suite import urllib as well (correctly). This might be very difficult to test.
I.e. it is probably ok to just fix it in the code.