Skip to content

easy options string storage - #22628

Closed
icing wants to merge 3 commits into
curl:masterfrom
icing:u8_strset
Closed

icing wants to merge 3 commits into
curl:masterfrom
icing:u8_strset

Conversation

@icing

@icing icing commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

easy option string storage

Change the storage of easy handle option strings from an array sized for
all possible options to a hash set to reduce memory footprint.

Give the hash set initially room for 4 strings, with first allocation
happening when it goes beyond that. Measurements with our test suite
and a forced fail on growing the set gives:

Size Result
2 ---> 1261 tests out of 1951 reported OK: 64%
4 ---> 1792 tests out of 1951 reported OK: 91%
8 ---> 1944 tests out of 1951 reported OK: 99%
16 ---> 1949 tests out of 1951 reported OK: 99%
32 ---> single fail of 3211, unit test for u8_strset

Add u8_strset that keeps the tuples (uint8_t id, char *str)
and allows set/unset by id. Add that as data->set.strings.

Define MACROS

  • CURL_EASY_STR(data, id) for access
  • CURL_EASY_STR_SET(data, id, s) for setting, making a copy
  • CURL_EASY_STR_SETN(data, id, s) for setting, no copy
  • CURL_EASY_STR_CLEAR(data, id) for unsetting
  • CURL_EASY_STR_CLEAR0(data, id) for unsetting and zero-ing value

Add data->set.str_copypostfields to handle former STRING_COPYPOSTFIELDS
string that was not always a string and could carry NUL bytes.

Add unit tests to test3211.

Change the storage of easy handle option strings from an array sized for
all possible options to a hash set to reduce memory footprint.

Give the hash set initially room for 4 strings, with first allocation
happening when it goes beyond that. Measurements without test suite
and a forced fail on growing the set gives:

Size Result
2    1261 tests out of 1951 reported OK: 64%
4    1792 tests out of 1951 reported OK: 91%
8    1944 tests out of 1951 reported OK: 99%
16   1949 tests out of 1951 reported OK: 99%
32   single fail of 3211, unit test for u8_strset

Add u8_strset that keeps the tuples (uint8_t id, char *str)
and allows set/unset by `id`. Add that as data->set.strings.

Define MACROS
* CURL_EASY_STR(data, id) for access
* CURL_EASY_STR_SET(data, id, s) for setting, making a copy
* CURL_EASY_STR_SETN(data, id, s) for setting, no copy
* CURL_EASY_STR_CLEAR(data, id) for unsetting
* CURL_EASY_STR_CLEAR0(data, id) for unsetting and zero-ing value

Add `data->set.str_copypostfields` to handle former `STRING_COPYPOSTFIELDS`
string that was not always a string and could carry NUL bytes.

Add unit tests to test3211.
@icing icing changed the title Use string set for config strings easy options string storage Aug 20, 2026
@icing
icing marked this pull request as ready for review August 20, 2026 14:20
@icing
icing requested a review from bagder August 20, 2026 14:40
@bagder
bagder requested a balanced review from Copilot August 20, 2026 20:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Replaces the easy handle’s fixed string array with a compact uint8_t-keyed hash set.

Changes:

  • Adds Robin Hood hash-set storage and access macros.
  • Migrates option handling and consumers to the new storage.
  • Separates binary COPYPOSTFIELDS storage and adds unit coverage.

Reviewed changes

Copilot reviewed 44 out of 44 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/unit/unit3211.c Tests the new string set.
tests/unit/unit1620.c Updates string cleanup assertions.
tests/data/test3211 Updates test description.
lib/vtls/wolfssl.c Migrates wolfSSL string access.
lib/vtls/vtls.c Migrates ECH access.
lib/vtls/vtls_config.c Migrates TLS configuration strings.
lib/vtls/schannel.c Migrates pinned-key access.
lib/vtls/rustls.c Migrates ECH strings.
lib/vtls/openssl.c Migrates OpenSSL option strings.
lib/vtls/mbedtls.c Migrates pinned-key access.
lib/vtls/gtls.c Migrates pinned-key access.
lib/vssh/vssh.c Migrates SSH key options.
lib/vssh/libssh2.c Migrates libssh2 options.
lib/vssh/libssh.c Migrates libssh options.
lib/vquic/vquic.c Migrates QUIC ECH access.
lib/vquic/vquic-tls.c Migrates QUIC TLS options.
lib/vquic/cf-quiche.c Migrates custom request access.
lib/vquic/cf-ngtcp2.c Migrates custom request access.
lib/vdns/doh.c Migrates DoH configuration strings.
lib/vdns/asyn-ares.c Migrates resolver options.
lib/urldata.h Defines hash-set storage and macros.
lib/url.c Initializes, clears, and reads the set.
lib/uint-hashset.h Declares the string hash set.
lib/uint-hashset.c Implements the string hash set.
lib/transfer.c Migrates transfer URL handling.
lib/smtp.c Migrates SMTP option strings.
lib/setopt.h Updates the string setter API.
lib/setopt.c Migrates option storage and post fields.
lib/rtsp.c Migrates RTSP option strings.
lib/proxy.c Migrates proxy option strings.
lib/pop3.c Migrates custom request access.
lib/Makefile.inc Adds hash-set sources.
lib/imap.c Migrates custom request access.
lib/http2.c Migrates HTTP/2 request access.
lib/http.c Migrates HTTP option strings.
lib/http_proxy.c Migrates proxy user-agent access.
lib/http_httpsig.c Migrates HTTP signature options.
lib/http_aws_sigv4.c Migrates AWS signature options.
lib/getinfo.c Migrates returned option strings.
lib/ftp.c Migrates FTP option strings.
lib/easy.c Duplicates hash-set and post-field storage.
lib/cookie.c Migrates cookie-jar access.
lib/cf-socket.c Migrates bind options.
lib/cf-haproxy.c Migrates HAProxy client IP access.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lib/urldata.h
Comment thread lib/uint-hashset.c
Comment thread tests/unit/unit3211.c
@bagder

bagder commented Aug 20, 2026

Copy link
Copy Markdown
Member

This PR shrinks the Curl_easy struct by almost 600 bytes...

icing added 2 commits August 21, 2026 09:05
- os400: fix old data->set.str[] use
- hashset: fix debug format
- unit3211: test hashset collisions and removals
@bagder bagder closed this in c8df3de Aug 21, 2026
vszakats added a commit that referenced this pull request Aug 24, 2026
…alls

Silencing:
```
In file included from libcurl_unity.c:179:
vssh/vssh.c: In function ‘Curl_ssh_setup_pkey.part.0’:
vssh/vssh.c:423:50: error: potential null pointer dereference [-Werror=null-dereference]
  423 |        CURL_EASY_STR(data, STRING_SSH_PUBLIC_KEY)[0]) {
libtool: compile:  gcc -DHAVE_CONFIG_H -I../include -I../lib -I../lib -I. -DBUILDING_LIBCURL -DCURL_STATICLIB -DUNITTESTS -D_GNU_SOURCE -Werror-implicit-functi
```
Ref: https://app.circleci.com/pipelines/gh/curl/curl/20825/workflows/c3eb591b-5a7d-40da-baa8-d2d9980b0cb2/jobs/181277

Follow-up to c8df3de #22628

Closes #22656
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Development

Successfully merging this pull request may close these issues.

3 participants