Do you want a change that makes URL parsing approximately 2 times faster?

[AyumuKasuga]

Hello. I looked at the speed of httpx2.URL(...), and I found that one line uses a large part of the parse time. I made a change for it, but I would like to know if you want this change before it goes further.

The contributing guide asks for an "Ideas" discussion first. This repository has only the "Announcements" and "Q&A" categories, so I write here. Please move this if there is a better place.

What I found

urlparse must reject the ASCII control characters that a URL cannot contain. To find them, it examines each character of the URL with a generator in Python:

if any(char.isascii() and not char.isprintable() for char in url):
    char = next(char for char in url if char.isascii() and not char.isprintable())
    idx = url.find(char)

A profile of httpx2.URL(...) shows that this generator uses approximately 43 percent of the parse time. The code also reads the string three times when the URL contains an incorrect character. urlparse does the same test for each keyword component.

What I suggest

The ASCII characters that are not printable are exactly the C0 control characters and DEL. One compiled regex search can replace the generator. The match object gives the character and also its position. The two additional passes are not necessary.

Two smaller changes to quote give the remainder of the increase. urlparse calls this function five times for each URL.

• Call PERCENT_ENCODED_REGEX.finditer(...) and not re.finditer(PERCENT_ENCODED_REGEX, ...). The second form repeats the re._compile cache operation at each call.
• Call percent_encoded immediately if the string contains no % character. Most strings do not contain this character.

The public API does not change. The error messages and the character positions do not change.

Measurements

The measurements use 40 rounds. Each variant runs in its own process. The order of the two variants changes at each round. The values are medians:

case	before	after	speed increase
short (25 characters)	4.61 us	2.57 us	1.79x
typical (69 characters)	7.42 us	3.64 us	2.04x
percent-encoded (70 characters)	7.85 us	4.91 us	1.60x
long query (1 KB)	30.31 us	12.28 us	2.47x

The two sets of samples do not overlap. In each case, the slowest of the 40 new samples is faster than the fastest of the 40 old samples. The increase is larger for a long URL, because the previous code reads each character in Python.

Is the change safe

The change is correct only if [\x00-\x1f\x7f] matches the same characters as the previous condition. Three tests show this:

• A test compares the two sets of characters for all 1114112 Unicode code points. The two sets are equal. Each set has 33 characters.
• A test compares the new code with the previous code for approximately 60000 fixed and random URLs. The results and the exception messages are the same.
• A test compares the character and the position that the code reports, for approximately 200000 strings.

The coverage stays at 100 percent.

My question

Pull request #1139 has the change, the tests and the benchmarks. I did not open a discussion first, and I am sorry about that.

Do you want this change? I am happy to make it smaller, to divide it into parts, or to close it.

[AyumuKasuga]

duplicate