-set_serial race condition can result in SEC_ERROR_REUSED_ISSUER_AND_SERIAL

[JohnnySheppard]

Describe the bug
I'm trying out the TLS Interception. I've created my cert and keys using the makefile, and run the proxy pointing to the certificate and keys using their full path. Some sites aren't loading properly and in the proxy.py output, I'm seeing this error a lot:

on_request_complete:278 - OSError when wrapping client

I'm looking at the TLS interception because I'm trying to write a plugin that will block certain url's based on their path, not just their domain - but I don't seem to be able to see the path component when the site is using TLS - and even when I've set the proxy up for TLS interception - I still can only see the domain part of the address in the proxy server. I'm assuming it's because something's not right (probably in the way I've set it up) or has something to do with that error.

It's very possible I've done something wrong in my set up, but I feel like I've followed the instructions.

To Reproduce
Steps to reproduce the behavior:

1. Run make ca-certificates
2. Run proxy --hostname 0.0.0.0 --ca-key-file /home/pi/proxy.py/ca-key.pem --ca-cert-file /home/pi/proxy.py/ca-cert.pem --ca-signing-key-file /home/pi/proxy.py/ca-signing-key.pem
3. Load https://www.sky.com in firefox on windows 10 setup to point to the proxy server manually.
4. See error

Expected behavior
I would expect the page to load fully and the error not to be listed. In my plugin that I've tried - but not listed in this example (and which may also just not be working), I would expect to be able to see the path component listed as something other than they url when I dump request.url).

Version information

• OS: Windos 10
• Browser Firefox
• Device: PC
• proxy.py Version 2.0.0 running on a Raspberry Pi 3B+ with Python 3.7.3

Additional Information
I'm seeing the OSError with and without my plugin. My plugin is in a very rough trying to work out how to see the path component stage at the moment, and is based on the FilterByUpstreamHostPlugin. I don't think it is the problem but I'll list it here for completeness:

#!/bin/python3

# -*- coding: utf-8 -*-
"""
    proxy.py
    ~~~~~~~~
    ⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
    Network monitoring, controls & Application development, testing, debugging.

    :copyright: (c) 2013-present by Abhinav Singh and contributors.
    :license: BSD, see LICENSE for more details.
"""
from typing import Optional


from proxy.http.exception import HttpRequestRejected
from proxy.http.parser import HttpParser
from proxy.http.codes import httpStatusCodes
from proxy.http.proxy import HttpProxyBasePlugin


class FilterByUpstreamHostPlugin2(HttpProxyBasePlugin):
    """Drop traffic by inspecting upstream host."""

    FILTERED_DOMAINS = [b'google.com',
                        b'www.google.com', b'www.bbc.co.uk/news']

    def before_upstream_connection(
            self, request: HttpParser) -> Optional[HttpParser]:
        print(request.url)
        if request.host in self.FILTERED_DOMAINS:
            raise HttpRequestRejected(
                status_code=httpStatusCodes.I_AM_A_TEAPOT, reason=b'I\'m a tea pot',
                headers={
                    b'Connection': b'close',
                }
            )
        return request

    def handle_client_request(
            self, request: HttpParser) -> Optional[HttpParser]:
        return request

    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
        return chunk

    def on_upstream_connection_close(self) -> None:
        pass

I really hope you can help. The proxy looks fantastic and will help me out loads with the project I'm on if I can get it to work.

Thanks in advance.

[abhinavsingh]

Hi @JohnnySheppard thanks for reporting this. I think you are running into similar issue as discussed here #234 (comment) @Benouare has posted a solution that worked out for him, can you try the same?

TL;DR - Makefile steps needs an update to use pki.py facility now.

[Benouare]

@JohnnySheppard, just a detail, do not forgot to add the ca-cert.pem in FF as CA certificates...

Edit : i just tried this sky.com with my own proxy ->

For gmc.com it works pretty good (requests that are blocked are not 'so' important in this case)

I think there is something else after that.

[Benouare]

Regarding that :

I tried to delete manualy the certs that looks wrong (with the red flag in FF) directly in my .proxy/certs/ dir. (rm -rf)
So, in the case of sky.com, I ve deleted static.skyassets.com.pem, after that, I reload the website. And directly after that, I ve deleted assets.sky.com.pem, and reload FF.
The result is that we want to :

First i thought that the pem was corrupted because of multiple thread, so i tried with 1 worker, and the issue remain (OSError & "duplicated certificate").

I will try to give u more informations this afternoon.

[abhinavsingh]

@Benouare This is already resolved now after your UUID fix in, correct? Wanted to double check before closing this out.