From e1658aa57e5ae6ded3549dbeb1ebebb311231343 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Sun, 6 Sep 2026 15:44:40 -0400 Subject: [PATCH 1/2] fix(hash): honor a custom comparator on legacy unsorted hash pages __ham_getindex_unsorted() does the linear search over a pre-4.6 P_HASH_UNSORTED page, which libdb still reads without requiring DB->upgrade. Its inline-key (H_KEYDATA) branch had two defects that masked each other whenever the application configured a comparator with DB->set_h_compare: 1. It called the comparator only to reject a non-zero result and never stored a zero one, so `res` kept its initial 1 (not-equal), the `if (res == 0) break;` after the switch never fired, and *match was set to 1 (not found). DB->get returned DB_NOTFOUND for a key that was present, and DB->put(DB_NOOVERWRITE) returned success and stored a second record with identical key bytes even though duplicates are disabled -- silently breaking key uniqueness. 2. It built the stored-key DBT with key->size, the length of the SEARCH key, instead of the length of the item actually on the page. The comparator therefore saw a truncated or over-long view of the stored key: a search key that merely prefixed a stored key compared equal (masked by defect 1 -- fixing only the dropped result turns the false negative into a false positive that returns another record's data), and an over-long search key made the comparator read past the stored item. Compare against the stored key at its own length and record the result, which is what __ham_getindex_sorted's equivalent case already does (itemlen = LEN_HKEYDATA(...), then res = t->h_compare(...)), and what the H_OFFPAGE case next door does by passing &res through to __db_moff. The memcmp path was always correct: it length-checks first and assigns its result. Only the comparator path was affected, so reproducing this needs a legacy page and an explicit DB->set_h_compare together -- a new handle's h_compare is NULL, and current sorted pages take __ham_getindex_sorted. Fixes #139 --- src/hash/hash_page.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/src/hash/hash_page.c b/src/hash/hash_page.c index f2c1ae3f0..27aceaacf 100644 --- a/src/hash/hash_page.c +++ b/src/hash/hash_page.c @@ -682,11 +682,15 @@ __ham_getindex_unsorted(dbc, p, key, match, indx) break; case H_KEYDATA: if (t->h_compare != NULL) { - DB_INIT_DBT(pg_dbt, - HKEYDATA_DATA(hk), key->size); - if (t->h_compare( - dbp, key, &pg_dbt) != 0) - break; + /* + * Compare against the stored key at its own + * length -- as __ham_getindex_sorted does -- + * and record the result: a 0 return means the + * keys are equal and the search is done. + */ + DB_INIT_DBT(pg_dbt, HKEYDATA_DATA(hk), + LEN_HKEY(dbp, p, dbp->pgsize, i)); + res = t->h_compare(dbp, key, &pg_dbt); } else if (key->size == LEN_HKEY(dbp, p, dbp->pgsize, i)) res = memcmp(key->data, HKEYDATA_DATA(hk), From a3f77db551d6985da55f615fd21001a79e5de38c Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Sun, 6 Sep 2026 15:44:59 -0400 Subject: [PATCH 2/2] test: regression gate for the unsorted-hash custom-comparator lookup test/db/hash_unsorted_cmp.c + run_hash_unsorted_cmp.sh cover the __ham_getindex_unsorted() comparator branch (issue #139), which no existing test reached: the branch needs a legacy P_HASH_UNSORTED page AND an explicitly configured DB->set_h_compare at the same time. test093 sets a comparator but only over current-format sorted pages (which take __ham_getindex_sorted), and run_upgrade.sh reads legacy pages but never sets a comparator. The legacy fixture is manufactured with the technique run_upgrade.sh already uses for old-format fixtures -- no old library and no committed binary blob. A P_HASH_UNSORTED page has the same byte layout as a P_HASH page; P_HASH merely additionally keeps its pairs in comparison order. So the driver creates a current-format Hash db (512-byte pages, inline and off-page keys), then rewrites each bucket page's PAGE.type byte from P_HASH to P_HASH_UNSORTED and the metadata version back to the 4.5.20 hash version 8. That is exactly the file __ham_getindex dispatches to the unsorted path. Checks, each run with the comparator (trigger) and without it (control) so a failure is attributable to the comparator path rather than to the fixture: * DB->get of a stored inline key succeeds and returns its value; * DB->get of a stored off-page key succeeds (the H_OFFPAGE branch next door, which passes &res to __db_moff, must stay correct); * DB->put(DB_NOOVERWRITE) over a live key returns DB_KEYEXIST and adds no record (verified by a full cursor scan); * a key that merely prefixes a stored key, and one that extends it, are both reported absent -- these catch the wrong-length stored-key DBT, including the false positive that a result-only fix would introduce. Against the unfixed library the four trigger checks fail (DB->get => DB_NOTFOUND for a stored key; DB_NOOVERWRITE => success with 22 records, 2 carrying the target key) while every control passes. Registered in the coverage subset (run_coverage.sh, full_run3_combined.sh) and documented in test/coverage/README.md. --- test/coverage/README.md | 21 ++ test/coverage/full_run3_combined.sh | 1 + test/coverage/run_coverage.sh | 10 + test/db/hash_unsorted_cmp.c | 550 ++++++++++++++++++++++++++++ test/db/run_hash_unsorted_cmp.sh | 57 +++ 5 files changed, 639 insertions(+) create mode 100644 test/db/hash_unsorted_cmp.c create mode 100644 test/db/run_hash_unsorted_cmp.sh diff --git a/test/coverage/README.md b/test/coverage/README.md index 69fefb5f2..87c219309 100644 --- a/test/coverage/README.md +++ b/test/coverage/README.md @@ -77,6 +77,27 @@ timeout, so they cannot hang: current db, because current off-page dups are already stored as a Recno tree (P_LRECNO/P_IRECNO), not a flat page chain. A genuine pre-3.1 fixture is required. +- **Legacy unsorted-hash lookup with a custom comparator** — + `test/db/run_hash_unsorted_cmp.sh` builds `test/db/hash_unsorted_cmp.c`, the + regression test for issue #139. `__ham_getindex_unsorted` + (`src/hash/hash_page.c`) is the linear search over a pre-4.6 + `P_HASH_UNSORTED` page; its `t->h_compare != NULL` inline-key branch was + never reached by any test, because the branch needs a legacy page **and** an + explicitly configured `DB->set_h_compare` at the same time (`test093` sets a + comparator but only over current-format sorted pages, which take + `__ham_getindex_sorted`; `run_upgrade.sh` reads legacy pages but never sets a + comparator). The branch called the comparator, dropped its result, and built + the stored-key DBT with the *search* key's length -- so an equal key was + reported absent (`DB->get` → `DB_NOTFOUND`, `DB_NOOVERWRITE` → success plus a + duplicate) and a search key that merely prefixed a stored key compared equal. + The driver manufactures its legacy fixture the same way `run_upgrade.sh` + does, without an old library or a committed blob: it creates a current-format + Hash db (small pages, inline + off-page keys), then rewrites each bucket + page's `PAGE.type` byte from `P_HASH` to `P_HASH_UNSORTED` and the metadata + version back to the 4.5.20 hash version 8 -- exactly what `__ham_getindex` + dispatches to the unsorted path. Every check runs twice, with and without the + comparator, so a failure is attributable to the comparator path and not to + the fixture. - **Async I/O backends (os_aio)** — `test/os/run_os_aio.sh` builds `test/os/os_aio_direct.c`, which links the internal libdb symbols and drives the async-I/O abstraction (`src/os/os_aio.c`) and its backends diff --git a/test/coverage/full_run3_combined.sh b/test/coverage/full_run3_combined.sh index a16c2bcbe..919bb3008 100644 --- a/test/coverage/full_run3_combined.sh +++ b/test/coverage/full_run3_combined.sh @@ -166,6 +166,7 @@ run_driver db_upgrade "$R/test/db/run_upgrade.sh" run_driver os_aio "$R/test/os/run_os_aio.sh" run_driver backup "$R/test/backup/run_backup_direct.sh" run_driver recd_compact "$R/test/db/run_recd_compact.sh" +run_driver hash_unsorted_cmp "$R/test/db/run_hash_unsorted_cmp.sh" log " .gcda after C drivers: $(find .libs -name '*.gcda' | wc -l)" # COV_DEAD_REG: deadlock detector + DB_REGISTER (driver-per-test) diff --git a/test/coverage/run_coverage.sh b/test/coverage/run_coverage.sh index 4f169f3e1..c4e4d5d72 100755 --- a/test/coverage/run_coverage.sh +++ b/test/coverage/run_coverage.sh @@ -347,6 +347,16 @@ if [ "${COV_BACKUP:-1}" = 1 ]; then else echo "FAIL recd_handlers (rc=$?)"; tail -5 /tmp/cov-recdhandlers.log fi + # Legacy P_HASH_UNSORTED lookup with a custom DB->set_h_compare: the + # h_compare branch of __ham_getindex_unsorted needs a pre-4.6 page AND an + # explicit comparator at the same time, which no Tcl test arranges + # (test093 sets a comparator over sorted pages; run_upgrade.sh reads legacy + # pages without one). Regression gate for issue #139. + if sh "$root/test/db/run_hash_unsorted_cmp.sh" >/tmp/cov-hashunsorted.log 2>&1; then + echo "PASS hash_unsorted_cmp" + else + echo "FAIL hash_unsorted_cmp (rc=$?)"; tail -5 /tmp/cov-hashunsorted.log + fi echo " .gcda files after backup/compact: $(find . -name '*.gcda' | wc -l)" fi diff --git a/test/db/hash_unsorted_cmp.c b/test/db/hash_unsorted_cmp.c new file mode 100644 index 000000000..c8d8a9a9d --- /dev/null +++ b/test/db/hash_unsorted_cmp.c @@ -0,0 +1,550 @@ +/*- + * See the file LICENSE for redistribution information. + * + * Copyright (c) 2026 Oracle and/or its affiliates. All rights reserved. + * + * hash_unsorted_cmp.c -- + * Regression driver for the Hash lookup defect reported as + * https://github.com/berkeleydb/libdb/issues/139: + * __ham_getindex_unsorted() (src/hash/hash_page.c) called the + * application's DB->set_h_compare comparator for an inline (H_KEYDATA) + * key but discarded its result, so an equal key on a legacy + * P_HASH_UNSORTED page was reported as absent. DB->get returned + * DB_NOTFOUND for a key that was present, and DB->put(DB_NOOVERWRITE) + * returned success and stored a second record with identical key bytes + * in a database where duplicates are disabled. + * + * The defect needs three things at once, which is why it hid: a Hash + * database with a page still in the pre-4.6 P_HASH_UNSORTED format (5.3 + * reads those without requiring DB->upgrade), an inline key on that + * page, and an explicitly configured custom comparator (a new handle's + * h_compare is NULL, so simply opening an old file is not enough). + * Modern sorted pages take __ham_getindex_sorted, which does record its + * comparison result. + * + * The legacy fixture is built synthetically -- no old library and no + * committed binary blob. A P_HASH_UNSORTED page and a P_HASH page have + * identical byte layouts; the only difference is that P_HASH keeps its + * key/data pairs in comparison order, which is a *subset* of what + * P_HASH_UNSORTED allows. So a current-format Hash file whose bucket pages + * have their PAGE.type byte (offset 25) rewritten from P_HASH (13) to + * P_HASH_UNSORTED (2), and whose metadata version (offset 16) is set back to + * the 4.5.20 hash version 8, is a legitimate legacy file: it is exactly what + * __ham_getindex dispatches to __ham_getindex_unsorted. (This is the same + * technique test/db/run_upgrade.sh uses to build old-format fixtures.) + * + * Checks, each run with the comparator (trigger) and without it (control), + * so a failure is attributable to the comparator path and not the fixture: + * 1. DB->get of a stored inline key must succeed and return its value. + * 2. DB->get of a stored off-page key must succeed (the H_OFFPAGE + * branch next door, which passes &res to __db_moff, must stay right). + * 3. DB->put(DB_NOOVERWRITE) for a stored key must return DB_KEYEXIST + * and must not add a record. + * 4. A search key that merely PREFIXES a stored key, and one that + * EXTENDS it, must both be reported absent. The old code built the + * stored-key DBT with the *search* key's length instead of the stored + * key's, so the comparator saw a truncated or over-long view of the + * page item: a prefix key compared equal (a false match returning + * another record's data), and an over-long key made the comparator + * read past the stored item. + * Before the fix, every "trigger" case failed; the controls passed. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "db.h" + +#define HOME "HASH_UNSORTED_TESTDIR" +#define PAGESIZE 512 /* Small pages: many bucket pages. */ +#define NRECS 20 +#define PAGE_TYPE_OFF 25 /* PAGE.type. */ +#define META_VERSION_OFF 16 /* DBMETA.version. */ +#define P_HASH_UNSORTED_T 2 /* Pre-4.6 hash page. */ +#define P_HASH_T 13 /* Sorted hash page. */ +#define HASH_VERSION_45 8 /* On-disk hash version of 4.5.20. */ +#define BIGKEYLEN 200 /* > PAGESIZE/4 => off-page key. */ +#define ALARM_SECS 120 + +static int fails = 0; +static unsigned long cmp_calls, cmp_equalities; + +#define CHK0(call) do { \ + int _r = (call); \ + if (_r != 0) { \ + fprintf(stderr, "FAIL: %s:%d: %s => %d (%s)\n", \ + __FILE__, __LINE__, #call, _r, db_strerror(_r)); \ + fails++; \ + } \ +} while (0) + +#define CHKEQ(got, want, what) do { \ + if ((got) != (want)) { \ + fprintf(stderr, "FAIL: %s:%d: %s: got %d, want %d\n", \ + __FILE__, __LINE__, (what), (int)(got), (int)(want));\ + fails++; \ + } \ +} while (0) + +/* + * byte_compare -- + * The comparison Berkeley DB itself used before 4.6 introduced + * DB->set_h_compare: unsigned byte order, shorter key first on a + * common prefix. DB->set_h_compare on an existing database requires + * exactly this -- a comparator that reproduces the ordering the + * database was created with. + */ +static int +byte_compare(DB *dbp, const DBT *a, const DBT *b) +{ + size_t len; + int ret; + + (void)dbp; + cmp_calls++; + len = a->size < b->size ? a->size : b->size; + if ((ret = memcmp(a->data, b->data, len)) == 0) + ret = a->size < b->size ? -1 : (a->size > b->size ? 1 : 0); + if (ret == 0) + cmp_equalities++; + return (ret); +} + +static void +inline_key(char *buf, int n) +{ + (void)snprintf(buf, 16, "acct%04d", n); +} + +static void +big_key(char *buf) +{ + memset(buf, 'K', BIGKEYLEN); + memcpy(buf, "offpage", 7); + buf[BIGKEYLEN] = '\0'; +} + +static void +set_dbt(DBT *dbt, void *data, u_int32_t size) +{ + memset(dbt, 0, sizeof(*dbt)); + dbt->data = data; + dbt->size = size; +} + +/* + * open_db -- + * Open the Hash database, optionally configuring the comparator. + */ +static int +open_db(DB **dbpp, const char *path, int custom, u_int32_t flags) +{ + DB *dbp; + int ret; + + if ((ret = db_create(&dbp, NULL, 0)) != 0) { + fprintf(stderr, "db_create: %s\n", db_strerror(ret)); + return (ret); + } + if (custom && (ret = dbp->set_h_compare(dbp, byte_compare)) != 0) { + fprintf(stderr, "set_h_compare: %s\n", db_strerror(ret)); + (void)dbp->close(dbp, 0); + return (ret); + } + if ((ret = dbp->open(dbp, + NULL, path, NULL, DB_HASH, flags, 0644)) != 0) { + fprintf(stderr, "open %s: %s\n", path, db_strerror(ret)); + (void)dbp->close(dbp, 0); + return (ret); + } + *dbpp = dbp; + return (0); +} + +/* + * produce -- + * Build a current-format Hash database with inline and off-page keys. + */ +static int +produce(const char *path) +{ + DB *dbp; + DBT key, data; + char kbuf[16], vbuf[32], bkbuf[BIGKEYLEN + 1]; + int i; + + (void)unlink(path); + if (db_create(&dbp, NULL, 0) != 0) + return (1); + CHK0(dbp->set_pagesize(dbp, PAGESIZE)); + CHK0(dbp->open(dbp, + NULL, path, NULL, DB_HASH, DB_CREATE | DB_EXCL, 0644)); + for (i = 0; i < NRECS; i++) { + inline_key(kbuf, i); + (void)snprintf(vbuf, sizeof(vbuf), "balance=%d", 100 * i); + set_dbt(&key, kbuf, (u_int32_t)strlen(kbuf)); + set_dbt(&data, vbuf, (u_int32_t)strlen(vbuf)); + CHK0(dbp->put(dbp, NULL, &key, &data, 0)); + } + big_key(bkbuf); + set_dbt(&key, bkbuf, (u_int32_t)BIGKEYLEN); + set_dbt(&data, "offpage-value", 13); + CHK0(dbp->put(dbp, NULL, &key, &data, 0)); + CHK0(dbp->close(dbp, 0)); + return (fails); +} + +/* + * make_legacy -- + * Rewrite every sorted hash page into the pre-4.6 unsorted page type + * and set the metadata version back to the 4.5.20 hash version. + * Returns the number of pages converted, -1 on error. + */ +static int +make_legacy(const char *path) +{ + struct stat sb; + off_t off; + u_int32_t pagesize, version; + int converted, fd; + u_int8_t type; + + if ((fd = open(path, O_RDWR)) < 0) { + perror("open fixture"); + return (-1); + } + if (fstat(fd, &sb) != 0) { + perror("fstat fixture"); + (void)close(fd); + return (-1); + } + if (pread(fd, &pagesize, sizeof(pagesize), 20) != + (ssize_t)sizeof(pagesize) || pagesize == 0 || + sb.st_size % (off_t)pagesize != 0) { + fprintf(stderr, "fixture is not page aligned\n"); + (void)close(fd); + return (-1); + } + version = HASH_VERSION_45; + if (pwrite(fd, &version, sizeof(version), META_VERSION_OFF) != + (ssize_t)sizeof(version)) { + perror("pwrite version"); + (void)close(fd); + return (-1); + } + converted = 0; + for (off = 0; off < sb.st_size; off += (off_t)pagesize) { + if (pread(fd, &type, 1, off + PAGE_TYPE_OFF) != 1) { + perror("pread page type"); + (void)close(fd); + return (-1); + } + if (type != P_HASH_T) + continue; + type = P_HASH_UNSORTED_T; + if (pwrite(fd, &type, 1, off + PAGE_TYPE_OFF) != 1) { + perror("pwrite page type"); + (void)close(fd); + return (-1); + } + converted++; + } + if (close(fd) != 0) { + perror("close fixture"); + return (-1); + } + return (converted); +} + +/* + * count_unsorted -- + * How many P_HASH_UNSORTED pages does the file still hold? + */ +static int +count_unsorted(const char *path) +{ + struct stat sb; + off_t off; + u_int32_t pagesize; + int count, fd; + u_int8_t type; + + if ((fd = open(path, O_RDONLY)) < 0 || fstat(fd, &sb) != 0 || + pread(fd, &pagesize, sizeof(pagesize), 20) != + (ssize_t)sizeof(pagesize) || pagesize == 0) { + perror("inspect fixture"); + if (fd >= 0) + (void)close(fd); + return (-1); + } + for (count = 0, off = 0; off < sb.st_size; off += (off_t)pagesize) + if (pread(fd, &type, 1, off + PAGE_TYPE_OFF) == 1 && + type == P_HASH_UNSORTED_T) + count++; + (void)close(fd); + return (count); +} + +static int +copy_file(const char *src, const char *dst) +{ + FILE *in, *out; + char buf[4096]; + size_t n; + + if ((in = fopen(src, "rb")) == NULL) + return (1); + if ((out = fopen(dst, "wb")) == NULL) { + (void)fclose(in); + return (1); + } + while ((n = fread(buf, 1, sizeof(buf), in)) > 0) + if (fwrite(buf, 1, n, out) != n) { + (void)fclose(in); + (void)fclose(out); + return (1); + } + (void)fclose(in); + return (fclose(out) != 0); +} + +/* + * count_records -- + * Total records, and how many carry the target key bytes. Read with + * the built-in comparison so the count is independent of the code + * under test. + */ +static void +count_records(const char *path, int *records, int *dups) +{ + DB *dbp; + DBC *dbc; + DBT key, data; + char kbuf[16]; + int ret; + + *records = *dups = 0; + if (open_db(&dbp, path, 0, DB_RDONLY) != 0) { + fails++; + return; + } + CHK0(dbp->cursor(dbp, NULL, &dbc, 0)); + inline_key(kbuf, 0); + memset(&key, 0, sizeof(key)); + memset(&data, 0, sizeof(data)); + while ((ret = dbc->get(dbc, &key, &data, DB_NEXT)) == 0) { + (*records)++; + if (key.size == strlen(kbuf) && + memcmp(key.data, kbuf, key.size) == 0) + (*dups)++; + } + if (ret != DB_NOTFOUND) + CHK0(ret); + CHK0(dbc->close(dbc)); + CHK0(dbp->close(dbp, 0)); +} + +/* + * check_get -- + * DB->get must find a key that is on the legacy page. + */ +static void +check_get(const char *path, int custom, int big) +{ + DB *dbp; + DBT key, data; + char kbuf[BIGKEYLEN + 1], vbuf[32]; + const char *label; + u_int32_t ksize; + int ret; + + label = custom ? "trigger" : "control"; + cmp_calls = cmp_equalities = 0; + if (count_unsorted(path) < 1) { + fprintf(stderr, + "FAIL: %s: no P_HASH_UNSORTED page left in %s\n", + label, path); + fails++; + return; + } + if (open_db(&dbp, path, custom, DB_RDONLY) != 0) { + fails++; + return; + } + if (big) { + big_key(kbuf); + ksize = BIGKEYLEN; + (void)snprintf(vbuf, sizeof(vbuf), "offpage-value"); + } else { + inline_key(kbuf, 0); + ksize = (u_int32_t)strlen(kbuf); + (void)snprintf(vbuf, sizeof(vbuf), "balance=0"); + } + set_dbt(&key, kbuf, ksize); + memset(&data, 0, sizeof(data)); + ret = dbp->get(dbp, NULL, &key, &data, 0); + printf(" get %s %s key: ret=%d (%s) cmp_calls=%lu equal=%lu\n", + label, big ? "off-page" : "inline", ret, + ret == 0 ? "success" : db_strerror(ret), cmp_calls, + cmp_equalities); + CHKEQ(ret, 0, "DB->get of a stored key"); + if (ret == 0 && (data.size != strlen(vbuf) || + memcmp(data.data, vbuf, data.size) != 0)) { + fprintf(stderr, "FAIL: %s: wrong value for stored key\n", + label); + fails++; + } + if (custom && cmp_calls == 0) { + fprintf(stderr, "FAIL: %s: comparator was never called\n", + label); + fails++; + } + CHK0(dbp->close(dbp, 0)); +} + +/* + * check_missing -- + * A key that is not stored must be reported absent. "prefix" asks for + * a proper prefix of a stored key, otherwise for a stored key with + * extra bytes appended. Both probe whether the stored-key DBT handed + * to the comparator carries the stored item's real length. + */ +static void +check_missing(const char *path, int custom, int prefix) +{ + DB *dbp; + DBT key, data; + char kbuf[32]; + const char *label; + u_int32_t ksize; + int ret; + + label = custom ? "trigger" : "control"; + cmp_calls = cmp_equalities = 0; + if (open_db(&dbp, path, custom, DB_RDONLY) != 0) { + fails++; + return; + } + inline_key(kbuf, 0); /* "acct0000" */ + if (prefix) + ksize = 4; /* "acct" */ + else { + (void)strncat(kbuf, "XXXX", sizeof(kbuf) - strlen(kbuf) - 1); + ksize = (u_int32_t)strlen(kbuf); + } + set_dbt(&key, kbuf, ksize); + memset(&data, 0, sizeof(data)); + ret = dbp->get(dbp, NULL, &key, &data, 0); + printf(" get %s %s key (%.*s): ret=%d (%s) cmp_calls=%lu equal=%lu\n", + label, prefix ? "prefix-of-stored" : "extends-stored", + (int)ksize, kbuf, ret, + ret == 0 ? "success" : db_strerror(ret), cmp_calls, + cmp_equalities); + CHKEQ(ret, DB_NOTFOUND, "DB->get of a key that is not stored"); + CHK0(dbp->close(dbp, 0)); +} + +/* + * check_nooverwrite -- + * DB->put(DB_NOOVERWRITE) for a key that is present must return + * DB_KEYEXIST and leave the record count alone. + */ +static void +check_nooverwrite(const char *base, int custom) +{ + DB *dbp; + DBT key, data; + char path[256], kbuf[16]; + const char *label; + int dups, records, ret; + + label = custom ? "trigger" : "control"; + (void)snprintf(path, sizeof(path), "%s/put_%s.db", HOME, label); + if (copy_file(base, path) != 0) { + fprintf(stderr, "FAIL: %s: cannot copy fixture\n", label); + fails++; + return; + } + cmp_calls = cmp_equalities = 0; + if (count_unsorted(path) < 1) { + fprintf(stderr, + "FAIL: %s: no P_HASH_UNSORTED page in the copy\n", label); + fails++; + return; + } + if (open_db(&dbp, path, custom, 0) != 0) { + fails++; + return; + } + inline_key(kbuf, 0); + set_dbt(&key, kbuf, (u_int32_t)strlen(kbuf)); + set_dbt(&data, "balance=999999", 14); + ret = dbp->put(dbp, NULL, &key, &data, DB_NOOVERWRITE); + CHK0(dbp->close(dbp, 0)); + count_records(path, &records, &dups); + printf(" put(DB_NOOVERWRITE) %s: ret=%d (%s) records=%d " + "with_target_key=%d cmp_calls=%lu equal=%lu\n", label, ret, + ret == 0 ? "success" : db_strerror(ret), records, dups, + cmp_calls, cmp_equalities); + CHKEQ(ret, DB_KEYEXIST, "DB->put(DB_NOOVERWRITE) over a live key"); + CHKEQ(records, NRECS + 1, "record count after DB_NOOVERWRITE"); + CHKEQ(dups, 1, "records carrying the target key"); +} + +int +main(int argc, char *argv[]) +{ + char base[256]; + int converted; + + (void)argc; + (void)argv; + + (void)signal(SIGALRM, SIG_DFL); + (void)alarm(ALARM_SECS); + + (void)mkdir(HOME, 0755); + (void)snprintf(base, sizeof(base), "%s/legacy.db", HOME); + (void)unlink(base); + + if (produce(base) != 0) { + fprintf(stderr, "FAIL: could not build the base database\n"); + return (EXIT_FAILURE); + } + if ((converted = make_legacy(base)) < 1) { + fprintf(stderr, + "FAIL: no hash page converted to P_HASH_UNSORTED\n"); + return (EXIT_FAILURE); + } + printf("legacy fixture: %s (%d page(s) -> P_HASH_UNSORTED, " + "hash version %d)\n", base, converted, HASH_VERSION_45); + + /* Controls first: they prove the fixture itself is sound. */ + check_get(base, 0, 0); + check_get(base, 0, 1); + check_missing(base, 0, 1); + check_missing(base, 0, 0); + check_nooverwrite(base, 0); + + /* Triggers: the same operations with a custom comparator. */ + check_get(base, 1, 0); + check_get(base, 1, 1); + check_missing(base, 1, 1); + check_missing(base, 1, 0); + check_nooverwrite(base, 1); + + if (fails != 0) { + fprintf(stderr, + "hash_unsorted_cmp: %d check(s) FAILED\n", fails); + return (EXIT_FAILURE); + } + printf("hash_unsorted_cmp: PASS\n"); + return (EXIT_SUCCESS); +} diff --git a/test/db/run_hash_unsorted_cmp.sh b/test/db/run_hash_unsorted_cmp.sh new file mode 100644 index 000000000..2103b988a --- /dev/null +++ b/test/db/run_hash_unsorted_cmp.sh @@ -0,0 +1,57 @@ +#!/bin/sh - +# +# $Id$ +# +# run_hash_unsorted_cmp.sh -- +# Build and run hash_unsorted_cmp.c, the regression test for +# https://github.com/berkeleydb/libdb/issues/139 -- +# __ham_getindex_unsorted() (src/hash/hash_page.c) called the +# application's DB->set_h_compare comparator for an inline key on a +# legacy P_HASH_UNSORTED page but dropped its result, so an equal key +# was reported as absent: DB->get returned DB_NOTFOUND for a key that +# was present and DB->put(DB_NOOVERWRITE) returned success and stored a +# second record with identical key bytes in a no-duplicates database. +# +# The driver builds its own legacy fixture (no old library, no committed +# binary blob): it creates a current-format Hash database, then rewrites +# each bucket page's PAGE.type byte from P_HASH to P_HASH_UNSORTED and +# the metadata version back to the 4.5.20 hash version 8 -- the same +# technique test/db/run_upgrade.sh uses for its old-format fixtures. +# Each check runs twice, with and without the comparator, so a failure +# is attributable to the comparator path rather than to the fixture. +# +# Usage (from build_unix): +# sh ../test/db/run_hash_unsorted_cmp.sh +# +# Exits non-zero on failure or hang. + +set -e + +BUILD=${BUILD:-.} +SRC=${SRC:-../test/db/hash_unsorted_cmp.c} +HOME_DIR=${HOME_DIR:-HASH_UNSORTED_TESTDIR} +TIMEOUT=${TIMEOUT:-180} + +LIB="$BUILD/.libs/libdb-5.3.so" +if [ ! -f "$LIB" ]; then + LIB=$(ls "$BUILD"/.libs/libdb-*.so 2>/dev/null | head -1) +fi +[ -n "$LIB" ] || { echo "FAIL: libdb .so not found in $BUILD/.libs"; exit 1; } + +echo "Compiling hash_unsorted_cmp against $LIB" +gcc -g -O1 ${CFLAGS:-} -I"$BUILD" "$SRC" "$LIB" \ + -lpthread -Wl,-rpath,"$(cd "$BUILD/.libs" && pwd)" \ + -o "$BUILD/hash_unsorted_cmp" + +rm -f "$HOME_DIR"/*.db 2>/dev/null || true +mkdir -p "$HOME_DIR" + +echo "Running hash_unsorted_cmp (timeout ${TIMEOUT}s)" +if timeout "$TIMEOUT" "$BUILD/hash_unsorted_cmp"; then + echo "run_hash_unsorted_cmp.sh: PASS" + exit 0 +else + rc=$? + echo "run_hash_unsorted_cmp.sh: FAIL (rc=$rc)" + exit $rc +fi