Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError: '_RSAPrivateKey' object has no attribute 'sign' (Morango) #2805

Closed
benjaoming opened this issue Dec 4, 2017 · 19 comments
Closed
Labels
bug Behavior is wrong or broken changelog Important user-facing changes
Milestone

Comments

@benjaoming
Copy link
Contributor

benjaoming commented Dec 4, 2017

Observed behavior

Just built 0.7.0-beta5 and running it from the Debian, the setup process fails after completing the final step:

image

Expected behavior

Correctly create a key and quit.

User-facing consequences

Release blocker - or at least getting it explained is :)

Errors and logs

INFO 2017-12-04 17:16:34,287 cli Going to daemon mode, logging to /var/kolibri/.kolibri/server.log
ERROR 2017-12-04 17:18:26,979 base Internal Server Error: /api/deviceprovision/
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/viewsets.py", line 87, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/device/api.py", line 31, in create
    data = serializer.save()
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/serializers.py", line 191, in save
    self.instance = self.create(validated_data)
  File "/usr/lib/python3/dist-packages/kolibri/core/device/serializers.py", line 57, in create
    facility = Facility.objects.create(**validated_data.pop('facility'))
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 401, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 969, in save
    super(Facility, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 706, in save
    super(Collection, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/mptt/models.py", line 1001, in save
    super(MPTTModel, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 163, in save
    self.ensure_dataset()
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 975, in ensure_dataset
    super(Facility, self).ensure_dataset(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 175, in ensure_dataset
    inferred_dataset = self.infer_dataset(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 980, in infer_dataset
    self.dataset = FacilityDataset.objects.create()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 401, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/models.py", line 279, in save
    super(SyncableModel, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/utils/uuids.py", line 117, in save
    self.id = self.calculate_uuid()
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/models.py", line 324, in calculate_uuid
    self._morango_source_id = self.calculate_source_id()
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 113, in calculate_source_id
    self._morango_source_id = Certificate.generate_root_certificate("full-facility").id
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 81, in generate_root_certificate
    cert.sign_certificate(cert)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 120, in sign_certificate
    cert_to_sign.signature = self.sign(cert_to_sign.serialized)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 150, in sign
    return self.private_key.sign(value)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/crypto.py", line 55, in sign
    signature = self._sign(message)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/crypto.py", line 244, in _sign
    return self._private_key.sign(message, crypto_padding.PKCS1v15(), crypto_hashes.SHA256())
AttributeError: '_RSAPrivateKey' object has no attribute 'sign'
kolibri shell
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import morango
>>> morango.__version__
'0.0.2'
>>> try:
...     from M2Crypto import RSA as M2RSA
...     from M2Crypto import BIO as M2BIO
...     M2CRYPTO_EXISTS = True
... except:
...     M2CRYPTO_EXISTS = False
... 
>>> M2CRYPTO_EXISTS
False
>>> import cryptography
>>> cryptography.__file__
'/usr/lib/python3/dist-packages/cryptography/__init__.py'
>>> import cryptography
>>> cryptography.__version__
'1.2.3'

The C extensions are built....

$ ls /usr/lib/python3/dist-packages/kolibri/dist/cext/cp35/linux-x86_64/
asn1crypto  cffi  _cffi_backend.so  cryptography  idna	pycparser  pycparser-2.18.tar.gz  six.py

Steps to reproduce

I can release/upload the .deb installer with the error if necessary...

Context

  • Kolibri version: 0.7.0-beta5
  • Operating system: Ubuntu 16.04
  • Browser: ?
@benjaoming benjaoming added P0 - critical Priority: Release blocker or regression bug Behavior is wrong or broken labels Dec 4, 2017
@benjaoming benjaoming added this to the 0.7.0 milestone Dec 4, 2017
@benjaoming
Copy link
Contributor Author

I updated the description as I was debugging this.

It seems that there could be two separate issues:

  1. Morango doesn't work with Cryptography 1.2.3
  2. When C extensions kolibri/dist/cext are present, they are not added at the beginning of the sys.path

@benjaoming
Copy link
Contributor Author

Updated Morango's test matrix to include different versions of Cryptography: learningequality/morango#32

@benjaoming
Copy link
Contributor Author

Debugging suggests that it's not the way in which we append sys.path

kolibri@megaflop:~$ kolibri shell
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> sys.path
['/usr/bin', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3/dist-packages/kolibri/dist', '/usr/lib/python3/dist-packages/kolibri/dist/cext/cp35/linux-x86_64']

Rather, I think it's because this isn't done until line 150 in kolibri.utils.cli -- and the load order is really polluted by now.

These are of interest because they all load before the call to get_cext_path

try:
    from .build_config.default_settings import settings_path
except ImportError:
    settings_path = "kolibri.deployment.default.settings.base"

# ...

from django.core.management import call_command  # noqa

# ...

from kolibri.core.deviceadmin.utils import IncompatibleDatabase  # noqa

# ....

from . import server  # noqa
from .system import become_daemon  # noqa

# ....

@benjaoming
Copy link
Contributor Author

@lyw07 I will try to change where get_cext_path is loaded, do you recall any reason why it's added at this particular stage?

@lyw07
Copy link
Contributor

lyw07 commented Dec 4, 2017

No I don't think it was added at that place due to any reason. It was probably because I didn't know where to put it. Please feel free to change it! Thank you!

@benjaoming
Copy link
Contributor Author

@lyw07 to reproduce it, you need to install a wrong, system-wide cryptography, for instance python3-cryptography

@lyw07
Copy link
Contributor

lyw07 commented Dec 4, 2017

@benjaoming Thank you! I've reproduced the error on a debian VM and been waiting the buildkite's build. I'll update the status once I finish testing!

@benjaoming
Copy link
Contributor Author

@lyw07 I triggered a rebuild just 2 minutes ago :)

@indirectlylit indirectlylit removed this from the 0.7.0 milestone Dec 6, 2017
@indirectlylit indirectlylit removed the P0 - critical Priority: Release blocker or regression label Dec 6, 2017
@indirectlylit indirectlylit added this to the 0.7.1 milestone Dec 6, 2017
@benjaoming
Copy link
Contributor Author

Fixed in #2808

@benjaoming
Copy link
Contributor Author

This is still an issue because cryptography isn't bundled.

The path on Ubuntu 16.04 64-bit resolves to (in my current .deb test build) via kolibri.utils.env.get_cext_path:

/usr/lib/python3/dist-packages/kolibri/dist/cext/cp35/linux-x86_64

This path is empty :/ So the system version is picked once again, and Morango doesn't support it. See: learningequality/morango#37

@benjaoming
Copy link
Contributor Author

Here's a full traceback btw.. reproduced in 0.7.1-beta1 source dist built with Python 3:

ERROR 2018-01-28 23:15:47,287 base Internal Server Error: /api/deviceprovision/
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/viewsets.py", line 87, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/device/api.py", line 31, in create
    data = serializer.save()
  File "/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/serializers.py", line 191, in save
    self.instance = self.create(validated_data)
  File "/usr/lib/python3/dist-packages/kolibri/core/device/serializers.py", line 57, in create
    facility = Facility.objects.create(**validated_data.pop('facility'))
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 401, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 984, in save
    super(Facility, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 723, in save
    super(Collection, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/mptt/models.py", line 1001, in save
    super(MPTTModel, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 163, in save
    self.ensure_dataset()
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 990, in ensure_dataset
    super(Facility, self).ensure_dataset(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 175, in ensure_dataset
    inferred_dataset = self.infer_dataset(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 995, in infer_dataset
    self.dataset = FacilityDataset.objects.create()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 401, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/models.py", line 417, in save
    super(SyncableModel, self).save(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/utils/uuids.py", line 117, in save
    self.id = self.calculate_uuid()
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/models.py", line 465, in calculate_uuid
    self._morango_source_id = self.calculate_source_id()
  File "/usr/lib/python3/dist-packages/kolibri/auth/models.py", line 113, in calculate_source_id
    self._morango_source_id = Certificate.generate_root_certificate(FULL_FACILITY).id
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 88, in generate_root_certificate
    cert.sign_certificate(cert)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 135, in sign_certificate
    cert_to_sign.signature = self.sign(cert_to_sign.serialized)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/certificates.py", line 207, in sign
    return self.private_key.sign(value)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/crypto.py", line 55, in sign
    signature = self._sign(message)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/crypto.py", line 244, in _sign
    return self._private_key.sign(message, crypto_padding.PKCS1v15(), crypto_hashes.SHA256())
AttributeError: '_RSAPrivateKey' object has no attribute 'sign'

@lyw07 I think we did right in choosing to remove these 64 bit + MacOS versions of cryptography to significantly reduce the size of the source dist -- but it makes the issue all the more evident :)

@benjaoming
Copy link
Contributor Author

Duplicate: #3079

@lyw07
Copy link
Contributor

lyw07 commented Jan 29, 2018

@benjaoming Sorry for not noticing the error while I was testing! I was testing the py2only PR using all the VMs with Python 2 and Python 3, but I guess at that time, my branch didn't have the 64-bit cryptography removed, so it looked fine. I think to resolve this issue, I can put the Linux 64-bit cryptography back? Do you think if this will be fine for the size of our source dist? Thank you!

@lyw07
Copy link
Contributor

lyw07 commented Jan 29, 2018

Since we are using only Python 3 for the Debian installer, I think it might be enough to only put back Python 3's Linux 64-bit cryptography packages back? Even if Debian users use the whl file and Python 2 to run Kolibri instead of the Debian installer, they will not get the error, with the premise that they don't have python-cryptography installed already in their computer. Do you think this will work? @benjaoming thanks!

@benjaoming
Copy link
Contributor Author

benjaoming commented Jan 29, 2018

@lyw07 I think my best argument for the inclusion of an accelerated version of cryptography would be Raspberry Pi platforms, but they're armf and not released on PyPi - as mentioned earlier, those builds are available in the Raspbian repos, so the true solution should come at a later stage, where we pin a hard dependency on python-cryptography in our debian package.

@benjaoming
Copy link
Contributor Author

@lyw07 I think it's okay to leave 64 bit out for now... not because it wouldn't be easier to just add it back.. but more because it would be a kind of symptom treatment of a fundamental issue, namely that Morango breaks when there's a cryptography version on the system that cannot sign data, as with 1.2.3.

@lyw07
Copy link
Contributor

lyw07 commented Jan 29, 2018

@benjaoming thank you for the clarification!

@benjaoming
Copy link
Contributor Author

@lyw07 - nvrmnd what I'm saying about ARM for RPi! Both arm-v6 and arm-v7 architectures are in fact included already, the script finds and downloads them fine! I just cannot see them on PyPi.

@benjaoming
Copy link
Contributor Author

Fixed in #3138 and learningequality/morango#46

@indirectlylit indirectlylit added the changelog Important user-facing changes label Feb 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Behavior is wrong or broken changelog Important user-facing changes
Projects
None yet
Development

No branches or pull requests

3 participants