Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLS Session Context refactoring #73

Merged
merged 16 commits into from
Oct 12, 2016
Merged

TLS Session Context refactoring #73

merged 16 commits into from
Oct 12, 2016

Conversation

alexmgr
Copy link
Collaborator

@alexmgr alexmgr commented Oct 4, 2016

I'm looking at adding TLS 1.3 to scapy_ssl_tls. But before that, I think that TLSSessionCtx needs some refactoring.
It's quite difficult to integrate new things now. Here is my grand masterplan ;). @tintinweb, let me know if that works for you:

  1. Add an asymmetrical keystore to hold RSA/DSA keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of self.crypto.client.rsa.privkey and co to use the keystore. Mostly done
  2. Introduce a key exchange keystore to hold EDH/ECDH/... keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of crypto.server.dh.p and co to use the keystore.
  3. Introduce a symmetrical keystore to hold generated keys. tls_ctx will hold a reference to both a client and server keystore. Then get rid of crypto.session.key.client.mac and co to use the keystore.
  4. Introduce a TLSContext which will hold references to the above keystores. Per session, 2 TLS contexts will be created, one server side, one client side. Today we duplicate this through client/server distinction, but we can abstract this away
  5. TLSSessionCtx will bind 2 TLSContexts (client/server) together and continue to hold the general session info (master_secret, negotiated stuff). I think this will make Allow loading of session secrets directly #64 easier, and can merge at that point all the session serialization
  6. Finally refactor _process()

This has the advantage that all seperate components are in charge of loading/saving/printing as they which, and makes iterative changes (and UT) easier.

Let me know what you think. I'll try and conserve the external interfaces to minimize changes, but references to tls_ctx variables from outside will break.

- Introducing a keystore to abstract asymetrical keys from TLS
  context
- Encapsulated RSA utility functions into a new module:
  ssl_tls_keystore
- Added a test module to make keystore UT simpler and independant
  from crypto layer
- Refactored ssl_tls_crypto to use the asymetrical keystore layer,
  as well as various examples and tests
- Removed all references to rsa.pubkey and rsa.privkey. Everything
  uses the RSA keystore
- Moved TLSSessionCtx string templating to the RSAKeystore class.
  Each sub element can be responsible for it's formating. Looks
  weird right now, until I have completed the refactoring of other
  elements
- Use double quotes consistently
- Moved camel casing to lower_case_with_underscores casing
- Moved DH Kex parameter handling into the keystore
- Refactored the TLSSessionCtx to use the new DH keystore
- Refactored UT
@tintinweb
Copy link
Owner

definitely the way to go! that will make it pretty straight forward to implement the idea of #64 - so thats a big like from my side. I can probably review this next week as I am currently travelling an plan to release/push v1.2.3 to pypi by the end of this week (fixing lots of nasties)

- Made keystore naming consistent with kex_keystore
- Print the size of RSA keys in use
- UT needed to work with RSA keys for the size calculation to work
- Refactored the TLSSessionCtx to use the new ECDH keystore
- Client side and server side keys are maintained in seperate key
  stores
- TLScontext holds all crypto and session material for one end of
  the connection
- TLSSessionCtx holds a reference to 2 TLSContexts
@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 6, 2016

This was tedious, but I think it's in a decent state now.
Still need to finally refactor the _process() layer, but should be simple now.

- Preserved the stateless nature of __process. Just moved state
  handlers to seperate private functions
@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 10, 2016

New output from session ctx looks something like this:

TLS Session Context:
    negotiated.version: TLS_1_2
    negotiated.ciphersuite: ECDHE_RSA_WITH_AES_128_CBC_SHA256
    negotiated.key_exchange: ECDHE
    negotiated.encryption: ('AES', 16, 'CBC')
    negotiated.mac: SHA256
    negotiated.compression: NULL
    encrypted_premaster_secret: None
    premaster_secret: '\xc9\x83\xc6\x1d\x07\xc9\xc3\xa4\xcew\xce\x13\xcd\\\xd0\x8b\xd6\xc0\\\xee[\x96\xb1D\xf23T\xf0\xdaq\t\x1f'
    master_secret: 'j\x8b;\x83a\x97\xc9)\xd5PM\x00A\xfa\xae\x85\xeahN\xca\xe9\xd9=P[/\xf1\xefn(\xc1*\xe8\x96\xd8\xaa2\xdbs"\xa4\x89\x93\x02}\x84\x93\xdf'

    Client TLS context:
        random: 'W\xfc\x15`\xa8:\x03\xdb<\xb3:\x93\xc4\xf6\x9d\xdcFGe\xd3\xb0\xb1\xffR5\xe3I\x9a\x82\x90*\xdf'
        session_id: ''
        Empty Asymmetrical Keystore:
            certificate: None
            size: 0
            public: None
            private: None
        ECDH Keystore:
            curve: secp256r1
            size: 256
            public: (113275496704084431291487477677909195595759109595762060638950447132726474002456, 77954217625571679719663409439576132777386731898169341442210820337739525519775) on "secp256r1" => y^2 = x^3 + 115792089210356248762697446949407573530086143415290314195533631308867097853948x + 41058363725152142129326129780047268409114441015993725554835256314039467401291 (mod 115792089210356248762697446949407573530086143415290314195533631308867097853951)
            private: 64156282720036847036300497222265754557565650929814408823437749519510182577437
        ECDHE_RSA_WITH_AES_128_CBC_SHA256 Keystore:
            AES cipher:
                mode: CBC
                key: 'K\xca\xff\xf8\xbd\\"\xd6<`#\xa7&\xdc\x10\xe1'
                size: 128
                block_size: 128
                iv: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
            SHA256 hmac:
                key: '?\xae\x9f\x8d~\xc9h\x9e\xcb\xb8\xfd\x05"C(\xcb\xaa\xd4oS\xa5K2\x1e3\xf1qe\x0bz\x05\xca'
                size: 256

    Server TLS context:
        random: '\xe6\x7f\x84\x16\x81\x03e\x9f\xd8\xe6W\x12\xbb\x19\x97\x07\xb5\xd5Z\xde\xcf\x91\xaf\x87e\xf0\x92/%\x9f\xf8\xd9'
        session_id: '\x1a\xf2\xde\x0c\xe3\xf5\xb7\x84Fq\x063HS\xa3\xdf(iEkf\xf2".3\xf9y\x13]\xfc7\xef'
        RSA Keystore:
            certificate: '0\x82\x04n0\x82\x03V\xa0\x03\x02\x01\x02\x02\t\x00\xd1\xe1\xf5:\x92\x03%\x1a0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x05\x05\x000\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where0\x1e\x17\r150522191612Z\x17\r250519191612Z0\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcd{qe\xeeu(\xf1\x07\xcffn\xdcg>\xed\xc8cTN\xbe\x8c\xc3t\x13F\x01^\xea\x18*s\xa9\xe1\x8e&\xf6\xf1U=\x83\x84=+\xda\xcd\xd4P\x1f\xae\xc7\xb4\xf5Dk\x87\x90\x05?\x15.#\xf7\r\x12\x1c\xa7\xf6:"\xa6WSn\xe4\xb5\x0b\x87wV\x8e\xf4i\x90\\\xe0R\x11\x17\x8d\xd9\xeb\xe2#\xb2\x12F\xcc\xe4\xba\xf3Q\xd0\xb8\x1bFH0\xe1_\xb7\x17\x8c\xf5\xf3\x9evs\xdewy\xe5\xdb\xbdz=.\xa9\x85\x89\xb0\xd6\x0065Dv\x93\xed.\xc62\xc3\xdb\xb62\xac%N;\x8c\xd7\x8e\x1e\xa1`\x98&\'\xe2\xcd:6\x9cK\xb4<HaA\xb9\x7f\xbb\xd9\xd3\xcb\x01K\x92\xe0\xecn\xcfF\xde\xd6GI\xbb\xec\xfbo\x98\xd0\xd2\xf4Y\xd5\xcf\x00T\xa6R"\x80\xaf\x96\x1d\xfc\xbe\x16P\x93q\x80\xf4=\xec\xf2\xf8r[\x94\xee\xec\x10$\x8c\xdcp\xac\xadc\xbc\xc3\xcdSp\xd0\xdc\x0f<\xba\x8d6\x99\t\xc6\xb9\x17\xf2C\xe5\xe5\xbc\'\x02\x03\x01\x00\x01\xa3\x81\xe80\x81\xe50\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14<\xc2\xf7\xfc\x85\xdb\xbeK\x05f\xb3[\xdetD\x84C\x8a\xe8>0\x81\xb5\x06\x03U\x1d#\x04\x81\xad0\x81\xaa\x80\x14<\xc2\xf7\xfc\x85\xdb\xbeK\x05f\xb3[\xdetD\x84C\x8a\xe8>\xa1\x81\x86\xa4\x81\x830\x81\x801\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nSome-State1\x120\x10\x06\x03U\x04\x07\x13\tSome-city1\x150\x13\x06\x03U\x04\n\x13\x0cSome-company1\x100\x0e\x06\x03U\x04\x0b\x13\x07Some-OU1\x1f0\x1d\x06\x03U\x04\x03\x13\x16some-server.some.where\x82\t\x00\xd1\xe1\xf5:\x92\x03%\x1a0\x0c\x06\x03U\x1d\x13\x04\x050\x03\x01\x01\xff0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x05\x05\x00\x03\x82\x01\x01\x00\x01s\x8e)\x85i-\x829\xfb\x17\x95\xe6\xea\x07\x18u\\\xf1\x06\xcds\x9fq\x13\xaf\xd3\xa0t\xad\xd0\x7f\x98\x1b\x06\xf3K\x9d\xf3\xe1e\x8c\x153U\xc5\x06\x1b6\x9d`\xd3A\xebL\xce\xfd\xd9\x8dmg\x90\xbeI\x9c\xde\x8b\xd5p]\x1a\x8a\x89\xbb\x14\x15\x99\xf31\x99\x14\xf8S\x9e)HH\xc1\x068b\x18\xd8g\x9d\xa4k\xa9\n,\xe7Xre\xcbU\xd6\xa6)V\x9beX\x1e\xe2\xe8\x8d\xed&K\x81\xdf\xf1\xc1\x1e,Ur\x8e\xfe\x17\r\xfeOvpo\xbb\xda\x13{\x02\xe0\xfa\x98sU\xb0\xcf\xdb?\x867\xe3Ts\xe4\xa6\xec\xcd\xcb\xc2}U\xd1\xf9V\xa5\xf2\xc4T\xe97\xdfq\xd4.!\xd4]"tw\xe2`S\xb8\xbe\x00?\xa5\'tk\x16;=K\x9aX](`\xe5\x08\x0e\xd9s}L_\xa5\xa3.\xeeE\xa4\xe5m\x8a\x03T#Ia\x90\x84X\x0c\xc9\xc6\xc2[\x1a\xc7\xf3\x85KP\x14#\xea\xfd\xd3(\x96\xaf\x92\xce\x8c\xa6\x929G\xd7|'
            size: 2048
            public: <_RSAobj @0x104cc7ab8 n(2048),e>
            private: None
        ECDH Keystore:
            curve: secp256r1
            size: 256
            public: (57839194958121498880865767025301393551536204925944902627237688213575334463653, 44403111253405849644576056020789267189925076126762403782064647303951800520579) on "secp256r1" => y^2 = x^3 + 115792089210356248762697446949407573530086143415290314195533631308867097853948x + 41058363725152142129326129780047268409114441015993725554835256314039467401291 (mod 115792089210356248762697446949407573530086143415290314195533631308867097853951)
            private: None
        ECDHE_RSA_WITH_AES_128_CBC_SHA256 Keystore:
            AES cipher:
                mode: CBC
                key: "\x82\xc8\xec\xa6\t\xccl':E\xee_\xec\xa3(\xf5"
                size: 128
                block_size: 128
                iv: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
            SHA256 hmac:
                key: '\xed\xfcRn\x02L\x9co\x8f\x10\xb9X\xe5\x0fK\x8f\x94W\x1e\x81\xaa\xe3\xe2\xfcWW\xa2?\xde\\^\xa6'
                size: 256

Formatting should be super easy to tweak. Should also be much easier to refactor further.
It also allows to serialize/restore any crypto state (interface TBD).

I'll put this through a PEP8 formatter and call it a day. After painfully rebasing #72 on top of this ;)

- Replaced remaining single quotes with double quotes
- Normalize the resulting files with: autopep8 -a -a -i
- Added pep8 section to setup.cfg to allow usage of long lines
@tintinweb
Copy link
Owner

thats looking pretty!
I started to rebase this off master after merging #72 and had to postpone it due to some brainteaser merge conflicts but I could probably help with that today.

- Resolved conflicts from session resumption into the refactoring
  work
@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 11, 2016

Just saw your comment. Has this almost done yesterday, so pushing this after merge.
Should solve all conflicts.

I'll put the whole project through autopep8 and we can merge this.

@tintinweb, would you mind giving this a quick test? I think all is good, but just to make sure. I ran most of the examples, including the automata stuff, so this should probably ll be OK.

An integration script would be nice at some stage to, to test all the examples @ commit time. Will see if I can do this later.

- Ran "autopep8 -a -a -i -r ." on project. Formatting should be more
  standardized now
@tintinweb
Copy link
Owner

sure, will check if my scripts have any issues and report back.

I can also take care of the integration suite if you want. it's on my todo list for some time and I keep getting reminded on this when manually checking our scripts for upcoming releases.

@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 11, 2016

Sounds good. Let me know how this goes.

For the integration scripts, sounds good also. Feel free to send to master directly if you wish.

 TLSExtSignatureAndHashAlgorithm; hash_algorithm -> hash_alg; signature_algorithm -> sig_alg
fixes security_scanner SLOTH checks due to attrib name change in TLSExtSignatureAndHashAlgorithm
adds TLS_1_4
@tintinweb
Copy link
Owner

  • fixed an attrib name inconsistency (doc/layer/examples)
  • added TLS_1_4 magic

nosetests shows an "Unknown client key exchange" warning, will have to investigate.

@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 12, 2016

nosetests shows an "Unknown client key exchange" warning, will have to investigate

That's "normal". It was a happening in the background before. I just added an else clause to raise a warning when that happens. Didn't bother digging into the root cause though since it's not a result of the refactor.

- RSA key loading should happen in the TLS context, not the session
  context. Avoids the `if client` logic
- Added a warning for unknown server kex
@alexmgr
Copy link
Collaborator Author

alexmgr commented Oct 12, 2016

This concludes it for me ;). Feel free to merge if you're happy with the testing and changes.

- resumption property was printed as compression
@tintinweb
Copy link
Owner

this is supernice! finished testing. I also see the warning for the full_Rsa_connection_with_application_data example but I agree it does not look like a regression.
good work!

@tintinweb tintinweb merged commit ebc8219 into master Oct 12, 2016
@alexmgr alexmgr deleted the tls_ctx_refactoring branch October 12, 2016 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants