Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3031 replace pycryptopp #616

Merged
merged 86 commits into from Jul 11, 2019
Merged

3031 replace pycryptopp #616

merged 86 commits into from Jul 11, 2019

Conversation

heartsucker
Copy link
Contributor

@heartsucker heartsucker commented May 17, 2019

Fixes: ticket #3031

Ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3031


This change is Reviewable

@meejah
Copy link
Contributor

meejah commented May 20, 2019

Hi @heartsucker just got back from the Summit and haven't taken an in-depth look yet.

I think cryptography is a great choice if it looks to cover the primitives we need! I do, however, feel like we've been lately moving towards more "factory functions" instead of "do complex things in constructors" -- so this would point to keeping something like make_keypair in favour of static/class methods. I would personally very much like to avoid a "wrapper" class if possible, as it makes the code harder to read for e.g. someone versed with cryptography.io already -- do you think it's feasible to use the "cryptography" objects directly, without wrapping (after creating them via some kind of factory-function in tahoe)?

@codecov-io
Copy link

codecov-io commented May 24, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@a07426b). Click here to learn what that means.
The diff coverage is 77.44%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #616   +/-   ##
=========================================
  Coverage          ?   84.97%           
=========================================
  Files             ?      162           
  Lines             ?    28852           
  Branches          ?     4107           
=========================================
  Hits              ?    24516           
  Misses            ?     3629           
  Partials          ?      707
Impacted Files Coverage Δ
src/allmydata/web/storage.py 96.64% <ø> (ø)
src/allmydata/_auto_deps.py 100% <ø> (ø)
src/allmydata/util/fileutil.py 50.84% <100%> (ø)
src/allmydata/introducer/common.py 96.49% <100%> (ø)
src/allmydata/mutable/retrieve.py 95.91% <100%> (ø)
src/allmydata/immutable/upload.py 95.64% <100%> (ø)
src/allmydata/scripts/admin.py 74.5% <100%> (ø)
src/allmydata/mutable/publish.py 96.65% <100%> (ø)
src/allmydata/immutable/filenode.py 95.91% <100%> (ø)
src/allmydata/client.py 90.99% <100%> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a07426b...41dd143. Read the comment docs.

@heartsucker
Copy link
Contributor Author

do you think it's feasible to use the "cryptography" objects directly, without wrapping

Yeah, we could do this too. Part of the reason I wrapped them was to keep existing APIs and provide helpers to all the crypto code was in one place. I really don't like having crypto code scattered all around the codebase because it makes it harder to say "no really don't do this" with something like key formatting. But I can make changes around that after I can even get this PR to work in the first place. 🙃

@heartsucker heartsucker marked this pull request as ready for review May 28, 2019 14:36
Copy link
Contributor

@meejah meejah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, looks pretty straightforward to use cryptography -- yay!

A lot of newer Tahoe code has been preferring to move to "factory functions" instead of doing complex work in object constructors. This would point to keeping the make_keypair (or similar) function that returns some objects (or, if the API was kept the same, returns some encoded keys and different helper functions create required objects .. like parse_public_key() / parse_private_key() or similar).

I see that the main "wrapper" classes are mostly to keep the API similar to pycryptopp..? If we do keep them, I'd at least prefer to get rid of @classmethods (and use bare helpers instead) and they all need to be new-style classes and have docstrings for everything.

However, consider: if I'm reading code that's using the AES class, it gives me .process() and that's it. I then have to figure out what self._encryptor is, find out it's a Cipher, and look up that doc instead. Consider, instead, an approach like:

def create_aes_encryptor(key, iv=None):
    """
    returns an object capable of doing AES encryption for Tahoe-LAFS. This
    is CTR mode with an optional IV (16 bytes of zeros by default).
    """"
    # do all the checks that are in AES's constructor here, instead

def encrypt_aes_bytes(aes_encryptor, plaintext):
    """
    Using an encryption object created by `create_aes_encryptor` this will
    return bytes that are an encrypted version of the `plaintext` bytes
    """
    # confirm aes_encryptor is a correct object
    return aes_encryptor.update(plaintext)

This way, it doesn't matter what kind of object create_aes_encryptor returns: users of
the code will always be doing things via function calls from this encryption helper module. The object
returned from the factory-function could be a straight Cryptography object or it could be an instance of class AES(object): which just promises to have a .process(). Or it could be an internal-only wrapper
and the "only" way to get work done is via encrypt_aes_bytes().

I don't necessarily see a benefit to sticking with the pycryptopp interface -- mostly because process() doesn't tell me anything when reading the code. It's used in relatively few places, and dropping in encrypt_aes_bytes(enc, chunk) vs. .process() is pretty easy to review.

src/allmydata/client.py Show resolved Hide resolved
src/allmydata/crypto/__init__.py Outdated Show resolved Hide resolved
src/allmydata/crypto/aes.py Show resolved Hide resolved
src/allmydata/crypto/ed25519.py Show resolved Hide resolved
src/allmydata/crypto/ed25519.py Show resolved Hide resolved
src/allmydata/crypto/ed25519.py Outdated Show resolved Hide resolved
src/allmydata/crypto/rsa.py Outdated Show resolved Hide resolved
src/allmydata/test/test_crypto.py Outdated Show resolved Hide resolved
src/allmydata/test/test_crypto.py Outdated Show resolved Hide resolved
src/allmydata/test/test_crypto.py Outdated Show resolved Hide resolved
src/allmydata/crypto/rsa.py Outdated Show resolved Hide resolved
@pythonhacker
Copy link

Looking at this. Takes a while.

@meejah
Copy link
Contributor

meejah commented Jun 11, 2019

re @heartsucker

I really don't like having crypto code scattered all around the codebase because it makes it harder to say "no really don't do this" with something like key formatting

I think in this case there's approximately the same amount of "crypto code scattered" because there are wrapper-objects that get used in this PR as it stands -- and so the only change would be "use the cryptography-provided classes" versus "use our own custom wrappers that just call through anyway". The motivation to use the cryptography-provided ones is that they're documented and better-known than a custom wrapper.

@meejah
Copy link
Contributor

meejah commented Jun 12, 2019

Here is what I mean by "helper functions" versus wrapper classes: #620

@exarkun opinions?

@meejah
Copy link
Contributor

meejah commented Jun 13, 2019

I have switched the RSA interfaces over to a functional-style one; the ed25519 stuff needs a similar treatment and some of the other fixups in the review have not been accomplished yet.

@meejah meejah changed the title 3031 replace pycryptopp [WIP] 3031 replace pycryptopp Jun 13, 2019
@meejah
Copy link
Contributor

meejah commented Jun 26, 2019

Can someone kick the circle-ci/integration builder to see if that was a transient error? (Running tox -e integration locally a few times hasn't produced a failure for me).

@tpltnt
Copy link
Contributor

tpltnt commented Jun 26, 2019

@heartsucker can you resolve the merge conflicts?

@meejah
Copy link
Contributor

meejah commented Jun 26, 2019

@tpltnt I don't think heartsucker is working on this any longer (but, I will rebase it)

@meejah meejah mentioned this pull request Jun 26, 2019
@meejah
Copy link
Contributor

meejah commented Jun 26, 2019

Okay, I re-based this ... but then I thought, "what is GitHub going to do if I force-push this?" so I have started a new PR in case github freaks out and "does bad stuff" to all the comments and discussion in here ...

@meejah
Copy link
Contributor

meejah commented Jun 26, 2019

Okay, on this branch I did "git merge master" to resolve the same conflict a different way .. so whether we merge this or "the other PR", meh.

Copy link
Member

@exarkun exarkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Some comments inline. I think this is going to be the last pass from me. Please address the inline comments and then I'm happy to see this merged.

src/allmydata/crypto/aes.py Show resolved Hide resolved
src/allmydata/crypto/aes.py Outdated Show resolved Hide resolved
src/allmydata/crypto/aes.py Outdated Show resolved Hide resolved
src/allmydata/crypto/aes.py Show resolved Hide resolved
src/allmydata/crypto/ed25519.py Outdated Show resolved Hide resolved
src/allmydata/test/test_introducer.py Show resolved Hide resolved
src/allmydata/test/test_util.py Outdated Show resolved Hide resolved
src/allmydata/test/test_util.py Show resolved Hide resolved
src/allmydata/util/fileutil.py Outdated Show resolved Hide resolved
src/allmydata/util/hashutil.py Outdated Show resolved Hide resolved
@meejah
Copy link
Contributor

meejah commented Jul 8, 2019

Okay, I believe I have addressed all comments. When CI goes green, should be good for merge -- if anyone else would like to review, that would be awesome! Please leave a comment if you are so I don't merge it ...

@meejah meejah merged commit ede9fc7 into master Jul 11, 2019
@meejah meejah deleted the 3031-replace-pycryptopp branch July 11, 2019 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants