TLS Interception Cert Generation #362

abhinavsingh · 2020-06-07T12:01:05Z

Fixes [Ubuntu] Cannot use TLS interception #299 where TLS interception not working as expected on Ubuntu was reported
Closes SAN extension required for TLS interception certificate generation #261 where we previously attempted a similar fix

@Benouare @httpnotonly @ja8zyjits @roshanprince402 @tawmoto @whitespots Folks PTAL at this branch and give it a try. Please report if TLS interception is still an issue.

I personally tested it on MacOS where TLS interception was broken too.
I am using following flags at my end: proxy --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key ca-signing-key.pem --ca-file venv373/lib/python3.7/site-packages/certifi/cacert.pem --plugins proxy.plugin.CacheResponsesPlugin
CA certificates were generated using make ca-certificates.
You can omit --ca-file flag on Ubuntu.
New approach uses custom openssl.cnf so this should also address Ubuntu use cases. But I haven't yet given it a try on Ubuntu.

Please let me know.

Screenshot of TLS interception via Chrome on MacOS. As we can see, certificate was signed by custom CA (example.com).

whitespots · 2020-06-07T12:24:16Z

Summary:
The issue is still here..
In my attempt, you will see the custom plugin, but it's the same as with the CacheResponsesPlugin.

My steps:

Installing requirements
make ca-certificates
Running on Mac OS with --ca-file key

python3.8 -m proxy \
--hostname 0.0.0.0 \
--port 8899 \
--log-level d \
--enable-dashboard \
--num-workers 4 \
--plugins proxy.plugin.SendRequestData \
--ca-key-file ca-key.pem \
--ca-file ca-cert.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem

Output

KeyError: "<socket.socket fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 55121)> (FD 22) is already registered"
ssl.SSLError
Traceback (most recent call last):
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 349, in handle_readables
    upgraded_sock = plugin.on_request_complete()
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 271, in on_request_complete
    self.wrap_server()
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 424, in wrap_server
    self.server._conn = ctx.wrap_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 417, in run
    self.shutdown()
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 208, in shutdown
    self.flush()
  File "/Users/appsec/GIT/proxy.py/proxy/http/handler.py", line 261, in flush
    self.selector.register(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/selectors.py", line 517, in register
    key = super().register(fileobj, events, data)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/selectors.py", line 241, in register
    raise KeyError("{!r} (FD {}) is already registered"
KeyError: "<socket.socket fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 55124)> (FD 22) is already registered"
^C2020-06-07 16:18:03,978 - pid:7097 [I] shutdown:101 - Shutting down 4 workers
...

Running on Docker:
docker build -t pyproxy .
Output:

2020-06-07 12:23:16,565 - pid:13 [D] recv:63 - received 201 bytes from client
2020-06-07 12:23:16,578 - pid:13 [D] connect_upstream:456 - Connecting to upstream mail.ru:443
2020-06-07 12:23:16,632 - pid:13 [D] connect_upstream:461 - Connected to upstream mail.ru:443
2020-06-07 12:23:16,654 - pid:13 [D] _new_conn:226 - Starting new HTTP connection (1): sequencer:2020
2020-06-07 12:23:16,670 - pid:13 [D] _make_request:433 - http://sequencer:2020 "POST /record HTTP/1.1" 200 2
2020-06-07 12:23:16,782 - pid:13 [D] gen_ca_signed_certificate:367 - Generating public key /root/.proxy.py/certificates/mail.ru.pub
2020-06-07 12:23:16,790 - pid:13 [E] on_request_complete:282 - OSError when wrapping client
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 432, in wrap_client
    generated_cert = self.generate_upstream_certificate(
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 414, in generate_upstream_certificate
    self.gen_ca_signed_certificate(cert_file_path)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 368, in gen_ca_signed_certificate
    resp = gen_public_key(public_key_path=public_key_path, private_key_path=private_key_path,
  File "/usr/local/lib/python3.8/site-packages/proxy/common/pki.py", line 106, in gen_public_key
    return run_openssl_command(command, timeout)
  File "/usr/local/lib/python3.8/site-packages/proxy/common/pki.py", line 211, in run_openssl_command
    cmd = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'openssl'
2020-06-07 12:23:16,795 - pid:13 [D] flush:91 - flushed 39 bytes to client
2020-06-07 12:23:16,797 - pid:13 [I] access_log:325 - 172.23.0.1:47572 - CONNECT mail.ru:443 - 0 bytes - 233.02 ms
2020-06-07 12:23:16,801 - pid:13 [D] on_client_connection_close:187 - Closed server connection, has buffer False

whitespots · 2020-06-07T12:38:57Z

@abhinavsingh
An interesting thing.
Python3.6 & 3.7 do not raise SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
But in both cases, there is an issue with sockets.

2020-06-07 12:35:56,992 - pid:17 [D] connect_upstream:463 - Connected to upstream mail.ru:443
2020-06-07 12:35:57,009 - pid:17 [D] _new_conn:230 - Starting new HTTP connection (1): sequencer:2020
2020-06-07 12:35:57,018 - pid:17 [D] _make_request:442 - http://sequencer:2020 "POST /record HTTP/1.1" 200 2
2020-06-07 12:35:57,124 - pid:17 [D] gen_ca_signed_certificate:367 - Generating public key /root/.proxy.py/certificates/mail.ru.pub
2020-06-07 12:35:57,141 - pid:17 [E] run:415 - Exception while handling connection <socket.socket fd=11, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('172.23.0.3', 8899), raddr=('172.23.0.1', 48044)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/usr/local/lib/python3.6/site-packages/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/usr/local/lib/python3.6/site-packages/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/usr/local/lib/python3.6/site-packages/proxy/http/handler.py", line 349, in handle_readables
    upgraded_sock = plugin.on_request_complete()
  File "/usr/local/lib/python3.6/site-packages/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/usr/local/lib/python3.6/site-packages/proxy/http/proxy/server.py", line 433, in wrap_client
    cast(Dict[str, Any], self.server.connection.getpeercert()))
  File "/usr/local/lib/python3.6/site-packages/proxy/http/proxy/server.py", line 414, in generate_upstream_certificate
    self.gen_ca_signed_certificate(cert_file_path)
  File "/usr/local/lib/python3.6/site-packages/proxy/http/proxy/server.py", line 371, in gen_ca_signed_certificate
    assert(resp is True)
AssertionError
2020-06-07 12:35:57,142 - pid:17 [D] shutdown:230 - Client connection closed

abhinavsingh · 2020-06-07T12:48:55Z

@abhinavsingh
An interesting thing.
Python3.6 does now raise SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]

@whitespots Thanks for a quick try. I suspect --ca-file ca-cert.pem is causing issue for you. Per flag documentation (my apologies, I should have used CA bundle file here)

Since you are on MacOS, simply pip install certifi and then use appropriate path as I did for --ca-file flag. This is the CA bundle file (not same as --ca-key-file). This CA bundle file will look like this (a list of CA root certificates):

This flag is required on MacOS because installed Python no longer has access to root CA bundle installed on MacOS.

But in both cases, there is an issue with sockets.

We run into socket issue because wrap_client failed, essentially leaving us with same old FD (pre-tls-interception attempt), leading us into (FD 22) is already registered error.

abhinavsingh · 2020-06-07T12:54:24Z

Running on Docker:
docker build -t pyproxy .
Output:

2020-06-07 12:23:16,565 - pid:13 [D] recv:63 - received 201 bytes from client
2020-06-07 12:23:16,578 - pid:13 [D] connect_upstream:456 - Connecting to upstream mail.ru:443
2020-06-07 12:23:16,632 - pid:13 [D] connect_upstream:461 - Connected to upstream mail.ru:443
2020-06-07 12:23:16,654 - pid:13 [D] _new_conn:226 - Starting new HTTP connection (1): sequencer:2020
2020-06-07 12:23:16,670 - pid:13 [D] _make_request:433 - http://sequencer:2020 "POST /record HTTP/1.1" 200 2
2020-06-07 12:23:16,782 - pid:13 [D] gen_ca_signed_certificate:367 - Generating public key /root/.proxy.py/certificates/mail.ru.pub
2020-06-07 12:23:16,790 - pid:13 [E] on_request_complete:282 - OSError when wrapping client
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 432, in wrap_client
    generated_cert = self.generate_upstream_certificate(
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 414, in generate_upstream_certificate
    self.gen_ca_signed_certificate(cert_file_path)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 368, in gen_ca_signed_certificate
    resp = gen_public_key(public_key_path=public_key_path, private_key_path=private_key_path,
  File "/usr/local/lib/python3.8/site-packages/proxy/common/pki.py", line 106, in gen_public_key
    return run_openssl_command(command, timeout)
  File "/usr/local/lib/python3.8/site-packages/proxy/common/pki.py", line 211, in run_openssl_command
    cmd = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'openssl'
2020-06-07 12:23:16,795 - pid:13 [D] flush:91 - flushed 39 bytes to client
2020-06-07 12:23:16,797 - pid:13 [I] access_log:325 - 172.23.0.1:47572 - CONNECT mail.ru:443 - 0 bytes - 233.02 ms
2020-06-07 12:23:16,801 - pid:13 [D] on_client_connection_close:187 - Closed server connection, has buffer False

Docker container doesn't have openssl installed hence leading us into FileNotFoundError: [Errno 2] No such file or directory: 'openssl'. Good point, I never packaged Docker containers with openssl.

Let's use this current branch for verification:

git checkout interception-cert-generation
git pull origin interception-cert-generation
python -m proxy ...... flags ......

Once after verification I can make a new release for pip, docker and brew distributions.

whitespots · 2020-06-07T13:09:20Z

I really need this library and that's why I'm testing right now :)
Will test OpenSSL installation in docker.

abhinavsingh · 2020-06-07T13:13:34Z

I really need this library and that's why I'm testing right now :)
Will test OpenSSL installation in docker.

Thank you. When you get time please lemme know about:

If it works on MacOS (without docker)
If you can update Dockerfile to also install openssl and verify it work, that will be excellent. Feel free to send PR for Dockerfile updates and we'll make a release.

Cheers!!!

whitespots · 2020-06-07T13:33:04Z

Dockerfile is updated with

...
LABEL com.abhinavsingh.name="abhinavsingh/proxy.py" \
      com.abhinavsingh.description="    Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on \
        Network monitoring, controls & Application development, testing, debugging." \
      com.abhinavsingh.url="https://github.com/abhinavsingh/proxy.py" \
      com.abhinavsingh.vcs-url="https://github.com/abhinavsingh/proxy.py" \
      com.abhinavsingh.docker.cmd="docker run -it --rm -p 8899:8899 abhinavsingh/proxy.py"

COPY --from=builder /deps /usr/local

RUN apk update && apk add openssl-dev openssl
...

But there is a problem with connections

2020-06-07 13:25:05,210 - pid:13 [D] handle_readables:302 - Client is ready for reads, reading
2020-06-07 13:25:05,211 - pid:13 [D] recv:63 - received 201 bytes from client
2020-06-07 13:25:05,220 - pid:13 [D] connect_upstream:456 - Connecting to upstream mail.ru:443
2020-06-07 13:25:05,270 - pid:13 [D] connect_upstream:461 - Connected to upstream mail.ru:443
2020-06-07 13:25:05,381 - pid:13 [D] gen_ca_signed_certificate:367 - Generating public key /root/.proxy.py/certificates/mail.ru.pub
2020-06-07 13:25:05,402 - pid:13 [E] run:413 - Exception while handling connection <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.23.0.3', 8899), raddr=('172.23.0.1', 50084)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 349, in handle_readables
    upgraded_sock = plugin.on_request_complete()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 432, in wrap_client
    generated_cert = self.generate_upstream_certificate(
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 414, in generate_upstream_certificate
    self.gen_ca_signed_certificate(cert_file_path)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 371, in gen_ca_signed_certificate
    assert(resp is True)
AssertionError
2020-06-07 13:25:05,406 - pid:13 [D] shutdown:230 - Client connection closed
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 417, in run
    self.shutdown()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 208, in shutdown
    self.flush()
  File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 261, in flush
    self.selector.register(
  File "/usr/local/lib/python3.8/selectors.py", line 352, in register
    key = super().register(fileobj, events, data)
  File "/usr/local/lib/python3.8/selectors.py", line 241, in register
    raise KeyError("{!r} (FD {}) is already registered"
KeyError: "<socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.23.0.3', 8899), raddr=('172.23.0.1', 50084)> (FD 12) is already registered"
2020-06-07 13:25:05,469 - pid:13 [D] initialize:145 - Handling connection <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.23.0.3', 8899), raddr=('172.23.0.1', 50090)>
2020-06-07 13:25:05,511 - pid:13 [D] handle_readables:302 - Client is ready for reads, reading
2020-06-07 13:25:05,513 - pid:13 [D] recv:63 - received 201 bytes from client

abhinavsingh · 2020-06-07T13:36:23Z

@whitespots Thank you. I'll check back on docker usage. From logs it is clear that underlying openssl command failed. But will need more investigation to understand why.

Meanwhile, does it work without docker on MacOS for you?

whitespots · 2020-06-07T13:38:13Z

@whitespots Thank you. I'll check back on docker usage. From logs it is clear that underlying openssl command failed. But will need more investigation to understand why.

Meanwhile, does it work without docker on MacOS for you?

..No :(
But I need more time to investigate it
Launch

python -m proxy \
--hostname 0.0.0.0 \
--port 8899 \
--log-level d \
--enable-dashboard \
--num-workers 1 \
--ca-file venv/lib/python3.8/site-packages/certifi/cacert.pem \
--ca-key-file ca-key.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem

Output

2020-06-07 17:36:43,915 - pid:8497 [D] start_workers:78 - Started acceptor#0 process 8501
2020-06-07 17:36:43,916 - pid:8497 [I] start_workers:84 - Started 1 workers
OSError when wrapping client
Traceback (most recent call last):
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 436, in wrap_client
    self.client._conn = ssl.wrap_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1402, in wrap_socket
    context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:4012)
OSError when wrapping client
Traceback (most recent call last):
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 276, in on_request_complete
    self.wrap_client()
  File "/Users/appsec/GIT/proxy.py/proxy/http/proxy/server.py", line 436, in wrap_client
    self.client._conn = ssl.wrap_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1402, in wrap_socket
    context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:4012)

abhinavsingh · 2020-06-07T13:51:06Z

Hmmm, this is weird. I gave it a try with Python 3.8 on MacOS and worked fine for me. I am wondering if ssl.SSLError: [X509: KEY_VALUES_MISMATCH] key values mismatch error could be due to old certificates under ~/.proxy/certificates. Among other things KEY_VALUES_MISMATCH can indicate mismatch of public and private keys.

Try to delete all older certificates under ~/.proxy/certificates directory and try again.

whitespots · 2020-06-07T13:53:39Z

It's empty actually..
But rm -rf ~/.proxy/ does not give me any success

…e generation error on Ubuntu

…h/proxy.py into interception-cert-generation

abhinavsingh · 2020-06-07T14:25:04Z

Update for Ubuntu

Gave it a try on Ubuntu

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

and TLS interception worked fine from command line. I didn't get a chance to setup browser for it. Please give it a try and lemme know.

whitespots · 2020-06-07T15:55:14Z

It's ok on the Ubuntu VM. Confirmed with FF also :)
Now we need to place it to the Dockerfile properly

whitespots · 2020-06-07T20:18:29Z

UPD
Only CONNECT method in logs is working now..
I am checking the course of this change (all was ok for 10 minutes)

Really interesting error for you, @abhinavsingh:

2020-06-08 00:25:35,294 - pid:28446 [E] run:415 - Exception while handling connection <ssl.SSLSocket fd=54, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 43428)>
Traceback (most recent call last):
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 331, in handle_readables
    client_data)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/proxy/server.py", line 215, in on_client_data
    self.pipeline_request.parse(raw.tobytes())
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 176, in parse
    more, raw = self.process(raw)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 186, in process
    self.process_line(line)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 214, in process_line
    self.set_url(line[1])
IndexError: list index out of range

PS. FF browser on Ubuntu says, that connection is secured by example.com, but Mac OS does not. Even after reboot.

codecov · 2020-06-09T06:18:38Z

Codecov Report

Merging #362 into develop will increase coverage by 0.15%.
The diff coverage is 96.77%.

@@             Coverage Diff             @@
##           develop     #362      +/-   ##
===========================================
+ Coverage    81.40%   81.55%   +0.15%     
===========================================
  Files           72       72              
  Lines         2775     2798      +23     
===========================================
+ Hits          2259     2282      +23     
  Misses         516      516

Impacted Files	Coverage Δ
proxy/http/proxy/server.py	`79.35% <96.66%> (+1.84%)`	⬆️
proxy/common/version.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab08901...2a9b75b. Read the comment docs.

abhinavsingh · 2020-06-09T06:23:57Z

UPD
Only CONNECT method in logs is working now..
I am checking the course of this change (all was ok for 10 minutes)

Yes, for HTTPS connections log will only show CONNECT method.

Really interesting error for you, @abhinavsingh:

2020-06-08 00:25:35,294 - pid:28446 [E] run:415 - Exception while handling connection <ssl.SSLSocket fd=54, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 43428)>
Traceback (most recent call last):
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 331, in handle_readables
    client_data)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/proxy/server.py", line 215, in on_client_data
    self.pipeline_request.parse(raw.tobytes())
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 176, in parse
    more, raw = self.process(raw)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 186, in process
    self.process_line(line)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 214, in process_line
    self.set_url(line[1])
IndexError: list index out of range

This error is interesting. Looks like an invalid request was made. Specifically, for a HTTP/1.1 pipelined request, subsequent requests didn't receive a URL. Curious, if you can share request parameters so I can dig further into it.

PS. FF browser on Ubuntu says, that connection is secured by example.com, but Mac OS does not. Even after reboot.

That's interesting because I am getting success on MacOS. Screenshot attached in description of this PR is actually from MacOS. What version of MacOS are you using? Also did you trust CA root certificate in keychain?

abhinavsingh · 2020-06-09T06:36:42Z

@Benouare @httpnotonly @ja8zyjits @roshanprince402 @tawmoto @whitespots

Folks, looks like we have a way forward. So I'll go ahead and merge this PR. I have personally verified the changes on both MacOS (using CLI and Browser) and Ubuntu (CLI only). @whitespots helped us verify this fix on Ubuntu (using Browser).

A few things to resolve going forward:

@whitespots reported TLS interception on MacOS doesn't say trusted. Frankly, this is more of a non-proxy problem and can be resolved outside of proxy.py changes a.k.a. trusting root certificate is not something proxy.py can do.
Currently proxy.py docker image is not shipped with openssl and hence users cannot try TLS interception with docker. Possible solutions:
- Mount openssl into container so that it is available to proxy.py running within docker
- Package openssl with proxy.py docker container (which will increase our container size)

abhinavsingh · 2020-06-09T06:38:11Z

2020-06-08 00:25:35,294 - pid:28446 [E] run:415 - Exception while handling connection <ssl.SSLSocket fd=54, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('127.0.0.1', 8899), raddr=('127.0.0.1', 43428)>
Traceback (most recent call last):
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 405, in run
    teardown = self.run_once()
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 390, in run_once
    teardown = self.handle_events(readables, writables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 193, in handle_events
    teardown = self.handle_readables(readables)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/handler.py", line 331, in handle_readables
    client_data)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/proxy/server.py", line 215, in on_client_data
    self.pipeline_request.parse(raw.tobytes())
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 176, in parse
    more, raw = self.process(raw)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 186, in process
    self.process_line(line)
  File "/media/psf/Home/GIT/proxy.py/proxy/http/parser.py", line 214, in process_line
    self.set_url(line[1])
IndexError: list index out of range

@whitespots If this problem persists please open a new issue with some details. Would love to address this if it's a constant problem. Thank you.

abhinavsingh added 3 commits June 7, 2020 17:24

Use common.pki for interception certificate generation

e791f7f

Fix tests

606e487

Merge branch 'develop' into interception-cert-generation

2bc8643

abhinavsingh mentioned this pull request Jun 7, 2020

SAN extension required for TLS interception certificate generation #261

Closed

abhinavsingh mentioned this pull request Jun 7, 2020

[TLS Interception] WebSocket (wss) is not working #356

Closed

abhinavsingh added 2 commits June 7, 2020 19:52

Dont use certificate fields that we dont need, it leads to certificat…

e0c7678

…e generation error on Ubuntu

Merge branch 'interception-cert-generation' of github.com:abhinavsing…

7bbabbb

…h/proxy.py into interception-cert-generation

abhinavsingh added 2 commits June 9, 2020 11:25

Merge branch 'develop' into interception-cert-generation

a82523a

Prepare for v2.2.0

6c953e3

npm audit fix

2a9b75b

abhinavsingh merged commit e7aa8a2 into develop Jun 9, 2020

abhinavsingh deleted the interception-cert-generation branch June 9, 2020 06:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TLS Interception Cert Generation #362

TLS Interception Cert Generation #362

abhinavsingh commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

abhinavsingh commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020 •

edited

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

codecov bot commented Jun 9, 2020 •

edited

abhinavsingh commented Jun 9, 2020

abhinavsingh commented Jun 9, 2020

abhinavsingh commented Jun 9, 2020

TLS Interception Cert Generation #362

TLS Interception Cert Generation #362

Conversation

abhinavsingh commented Jun 7, 2020 • edited

whitespots commented Jun 7, 2020 • edited

whitespots commented Jun 7, 2020 • edited

abhinavsingh commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020 • edited

abhinavsingh commented Jun 7, 2020

whitespots commented Jun 7, 2020

abhinavsingh commented Jun 7, 2020

Update for Ubuntu

whitespots commented Jun 7, 2020 • edited

whitespots commented Jun 7, 2020 • edited

codecov bot commented Jun 9, 2020 • edited

Codecov Report

abhinavsingh commented Jun 9, 2020

abhinavsingh commented Jun 9, 2020

abhinavsingh commented Jun 9, 2020

abhinavsingh commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

whitespots commented Jun 7, 2020 •

edited

codecov bot commented Jun 9, 2020 •

edited