From 16b4bd760c549973b51d8c744f1e9c10b93a4f7c Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 13 Feb 2024 11:17:23 +0200 Subject: [PATCH 1/9] Start a v6 format draft doc --- docs/manual/format_v6.md | 132 +++++++++++++++++++++++++++++++++++++++ docs/manual/index.md | 1 + 2 files changed, 133 insertions(+) create mode 100644 docs/manual/format_v6.md diff --git a/docs/manual/format_v6.md b/docs/manual/format_v6.md new file mode 100644 index 0000000000..f07fc761c4 --- /dev/null +++ b/docs/manual/format_v6.md @@ -0,0 +1,132 @@ +--- +layout: default +title: rpm.org - RPM V6 Package format +--- +# V6 Package format DRAFT + +This document describes the RPM file format version 6, which is used +by RPM versions 6.x and with limitations, readable with 4.x. + +**THE PROPER WAY TO ACCESS THESE STRUCTURES IS THROUGH THE RPM LIBRARY!!** + +The RPM file format covers both source and binary packages. An RPM +package file is divided in 4 logical sections: + +``` +. Lead -- 96 bytes of "magic" and other info +. Signature -- collection of "digital signatures" +. Header -- holding area for all the package information (aka "metadata") +. Payload -- compressed archive of the file(s) in the package (aka "payload") +``` + +All applicaple integer quantities are stored in network byte order +(big-endian). When data is presented, the first number is the +byte number, or address, in hex, followed by the byte values in hex, +followed by character "translations" (where appropriate). +All padding must consist solely of zero valued bytes. + +## Lead + +The Lead is used for identifying RPM package files based on the "magic" +value. The rest of the data in the Lead is historical only and is not +used by RPM. + +The Lead is always 96 bytes long and starts with a four byte "magic" +[ 0xED, 0xAB, 0xEE, 0xDB ]. For v6 packages, the "version" field contains +4 for backwards compatibility reasons, and OS and architecture are zeros. +For further details, refer to the [Lead Format](format_lead.md) document. + +## Signature + +The Signature uses the same underlying [data structure](format_header.md) +as the Header. It's size is internally padded to a multiple of 8 bytes. + +The Signature consists of a single immutable header region denoted +by it's first tag of 62 (RPMTAG_HEADERSIGNATURES) which can be used +to identify it. + +The Signature can contain several tags of different types: + +Name | Tag | Header Type +--------------------|-------|------------ +HEADERSIGNATURES | 62 | BIN +DSA | 267 | BIN +RSA | 268 | BIN +SHA256 | 272 | STRING +FILESIGNATURES | 274 | STRING_ARRAY +VERITYSIGNATURES | 276 | STRING_ARRAY +VERITYSIGNATUREALGO | 277 | INT_32 +RESERVED | 999 | BIN + +All packages carry at least HEADERSIGNATURES, SHA256 and RESERVED tags. + +On digitally signed packages, one of RSA or DSA tags is present and +contains an OpenPGP signature on the Header. The RSA tag is used for +RSA signatures and the DSA tag is used for EcDSA signatures. + +Tags numbers above 999, including those of v3 (header+payload) signatures, +are considered illegal on v6 packages. + +In addition, a package may also have either IMA or fsverity signatures +on it's files. If present, these are in FILESIGNATURES and VERITYSIGNATURES +tags respectively. + +RESERVED is always the last tag in the Signature. It's used as a space +reservation for signatures and consists solely of zeros. + +Strict tag sorting is required, and no duplicate tags are permitted. + +In summary, the difference to v4 Signature is that obsolete cryptography +has been removed along with all size and payload relevant data, and space +reservation uses a different tag. Also, there are no tag number conflicts +between the Signature and the Header. + +## Header + +The Header contains all the information about a package: name, +version, file list, etc. It uses the same underlying +[data structure](format_header.md) as the Signature. + +The Header consists of a single immutable header region denoted +by it's first tag of 63 (RPMTAG_HEADERIMMUTABLE) which can be used +to identify it. + +Strict tag sorting is required, and no duplicate tags are permitted. + +The complete list of tags is documented [here](tags.md), but in particular +for v6: +- New numeric tag RPMTAG_RPMFORMAT for the rpm package format version +- RPMTAG_ENCODING is required to be present and contain "utf-8" +- RPMTAG_LONGFILESIZES are always used to represent file sizes +- RPMTAG_FILEDIGESTALGO is always present and at least SHA256 in strength +- RPMTAG_PAYLOADDIGEST and RPMTAG_PAYLOADDIGESTALT are always present + and contain SHA256 hashes of the Payload (compressed and + uncompressed). Thus a header signature is sufficient to establish + cryptographic provenance of the package, without having to separately + calculate the payload signature. The alternatives allow freely switching + between a compressed and uncompressed payload for a package. +- Similarly, there are two alternative size tags on the Payload (compressed + and uncompressed): RPMTAG_PAYLOADSIZE and RPMTAG_PAYLOADSIZEALT +- File type information is stored as MIME types instead of libmagic + strings in RPMTAG_MIMEDICT and RPMTAG_FILEMIMEINDEX, retrievable + with RPMTAG_FILEMIMES extension. + +## Payload + +The payload is a stripped-down version of cpio, using `07070X` as magic bytes. +The file header only contains the index number of the file in the RPM +header as an 8 byte hex string. The payload may be compressed. + +Note: This is the format v4 uses for handling > 4GB files. + +## Differences to V4 + +The main differences of the V4 package format are: +- Cryptographic algorithms have been updated to contemporary standards +- Signature only carries cryptographic content +- Signature tags no longer clash with other tags +- All sizes are stored as 64bit integers (ie the LONG-variants size tags) +- Payload is verifiable independently of the other package components +- Payload is always in the "new" rpm specific format which has no size limits +- Strings are always UTF-8 encoded + diff --git a/docs/manual/index.md b/docs/manual/index.md index 15808f1112..39abe32b88 100644 --- a/docs/manual/index.md +++ b/docs/manual/index.md @@ -42,6 +42,7 @@ title: rpm.org - RPM Reference Manual * [Plugin API](plugins.md) ### Package Format +* [RPM v6 file format (DRAFT)](format_v6.md) * [RPM v4 file format](format_v4.md) * [RPM v4 signatures and digests](signatures_digests.md) * [RPM v3 file format](format_v3.md) (obsolete) From cd041129e86a4bb670da2175d7059dd6c847c9c8 Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Mon, 19 Feb 2024 11:13:44 +0200 Subject: [PATCH 2/9] Add tests for v6 package structure and digests, equal to v4 just now This may seem silly but it allows the test diffs to highlight the actual changes, making it easier to verify each change. --- tests/data/misc/rpmdump6.txt | 400 +++++++++++++++++++++++++++++++++++ tests/pinned/rpmsigdig6.sh | 5 + tests/pinned/rpmsigdig6.txt | 11 + tests/rpmpkgfmt.at | 17 ++ tests/rpmsigdig.at | 3 +- 5 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 tests/data/misc/rpmdump6.txt create mode 100755 tests/pinned/rpmsigdig6.sh create mode 100644 tests/pinned/rpmsigdig6.txt diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt new file mode 100644 index 0000000000..eeac859b21 --- /dev/null +++ b/tests/data/misc/rpmdump6.txt @@ -0,0 +1,400 @@ +Lead magic: edabeedb +Version 3.0 +Type: 0 +Name: attrtest-1.0-1 +Arch: 1 +OS: 1 +Sigtype: 5 + +Signature: +Header magic: 1e8ad8e (reserved: 0) +Index entries: 7 (112 bytes) +Data size: 4276 bytes +Header size: 4388 bytes +Padding: 4 bytes +Region entries 7 +Region size 128 +Dribbles: 0 + +Tag #0 [region] + tagno: 62 (Headersignatures) + type: 7 (blob) + offset: 4260 + count: 16 + + region trailer + tagno: 62 (Headersignatures) + type: 7 (blob) + offset: -112 + count: 16 + +Tag #1 [region] + tagno: 269 (Sha1) + type: 6 (string) + offset: 0 + count: 1 + +Tag #2 [region] + tagno: 273 (Sha256) + type: 6 (string) + offset: 41 + count: 1 + +Tag #3 [region] + tagno: 1000 (Size) + type: 4 (int32) + offset: 108 + count: 1 + +Tag #4 [region] + tagno: 1004 (Md5) + type: 7 (blob) + offset: 112 + count: 16 + +Tag #5 [region] + tagno: 1007 (Payloadsize) + type: 4 (int32) + offset: 128 + count: 1 + +Tag #6 [region] + tagno: 1008 (Reservedspace) + type: 7 (blob) + offset: 132 + count: 4128 + +Header: +Header magic: 1e8ad8e (reserved: 0) +Index entries: 53 (848 bytes) +Data size: 2573 bytes +Header size: 3421 bytes + +Region entries 53 +Region size 864 +Dribbles: 0 + +Tag #0 [region] + tagno: 63 (Headerimmutable) + type: 7 (blob) + offset: 2557 + count: 16 + + region trailer + tagno: 63 (Headerimmutable) + type: 7 (blob) + offset: -848 + count: 16 + +Tag #1 [region] + tagno: 100 (Headeri18ntable) + type: 8 (argv) + offset: 0 + count: 1 + +Tag #2 [region] + tagno: 1000 (Name) + type: 6 (string) + offset: 2 + count: 1 + +Tag #3 [region] + tagno: 1001 (Version) + type: 6 (string) + offset: 11 + count: 1 + +Tag #4 [region] + tagno: 1002 (Release) + type: 6 (string) + offset: 15 + count: 1 + +Tag #5 [region] + tagno: 1004 (Summary) + type: 9 (i18nstring) + offset: 17 + count: 1 + +Tag #6 [region] + tagno: 1005 (Description) + type: 9 (i18nstring) + offset: 44 + count: 1 + +Tag #7 [region] + tagno: 1006 (Buildtime) + type: 4 (int32) + offset: 72 + count: 1 + +Tag #8 [region] + tagno: 1007 (Buildhost) + type: 6 (string) + offset: 76 + count: 1 + +Tag #9 [region] + tagno: 1009 (Size) + type: 4 (int32) + offset: 88 + count: 1 + +Tag #10 [region] + tagno: 1014 (License) + type: 6 (string) + offset: 92 + count: 1 + +Tag #11 [region] + tagno: 1016 (Group) + type: 9 (i18nstring) + offset: 96 + count: 1 + +Tag #12 [region] + tagno: 1021 (Os) + type: 6 (string) + offset: 104 + count: 1 + +Tag #13 [region] + tagno: 1022 (Arch) + type: 6 (string) + offset: 110 + count: 1 + +Tag #14 [region] + tagno: 1028 (Filesizes) + type: 4 (int32) + offset: 120 + count: 20 + +Tag #15 [region] + tagno: 1030 (Filemodes) + type: 3 (int16) + offset: 200 + count: 20 + +Tag #16 [region] + tagno: 1033 (Filerdevs) + type: 3 (int16) + offset: 240 + count: 20 + +Tag #17 [region] + tagno: 1034 (Filemtimes) + type: 4 (int32) + offset: 280 + count: 20 + +Tag #18 [region] + tagno: 1035 (Filedigests) + type: 8 (argv) + offset: 360 + count: 20 + +Tag #19 [region] + tagno: 1036 (Filelinktos) + type: 8 (argv) + offset: 1020 + count: 20 + +Tag #20 [region] + tagno: 1037 (Fileflags) + type: 4 (int32) + offset: 1040 + count: 20 + +Tag #21 [region] + tagno: 1039 (Fileusername) + type: 8 (argv) + offset: 1120 + count: 20 + +Tag #22 [region] + tagno: 1040 (Filegroupname) + type: 8 (argv) + offset: 1218 + count: 20 + +Tag #23 [region] + tagno: 1044 (Sourcerpm) + type: 6 (string) + offset: 1307 + count: 1 + +Tag #24 [region] + tagno: 1045 (Fileverifyflags) + type: 4 (int32) + offset: 1332 + count: 20 + +Tag #25 [region] + tagno: 1047 (Providename) + type: 8 (argv) + offset: 1412 + count: 1 + +Tag #26 [region] + tagno: 1048 (Requireflags) + type: 4 (int32) + offset: 1424 + count: 15 + +Tag #27 [region] + tagno: 1049 (Requirename) + type: 8 (argv) + offset: 1484 + count: 15 + +Tag #28 [region] + tagno: 1050 (Requireversion) + type: 8 (argv) + offset: 1697 + count: 15 + +Tag #29 [region] + tagno: 1064 (Rpmversion) + type: 6 (string) + offset: 1731 + count: 1 + +Tag #30 [region] + tagno: 1080 (Changelogtime) + type: 4 (int32) + offset: 1740 + count: 1 + +Tag #31 [region] + tagno: 1081 (Changelogname) + type: 8 (argv) + offset: 1744 + count: 1 + +Tag #32 [region] + tagno: 1082 (Changelogtext) + type: 8 (argv) + offset: 1782 + count: 1 + +Tag #33 [region] + tagno: 1095 (Filedevices) + type: 4 (int32) + offset: 1800 + count: 20 + +Tag #34 [region] + tagno: 1096 (Fileinodes) + type: 4 (int32) + offset: 1880 + count: 20 + +Tag #35 [region] + tagno: 1097 (Filelangs) + type: 8 (argv) + offset: 1960 + count: 20 + +Tag #36 [region] + tagno: 1112 (Provideflags) + type: 4 (int32) + offset: 1980 + count: 1 + +Tag #37 [region] + tagno: 1113 (Provideversion) + type: 8 (argv) + offset: 1984 + count: 1 + +Tag #38 [region] + tagno: 1116 (Dirindexes) + type: 4 (int32) + offset: 1992 + count: 20 + +Tag #39 [region] + tagno: 1117 (Basenames) + type: 8 (argv) + offset: 2072 + count: 20 + +Tag #40 [region] + tagno: 1118 (Dirnames) + type: 8 (argv) + offset: 2162 + count: 10 + +Tag #41 [region] + tagno: 1122 (Optflags) + type: 6 (string) + offset: 2202 + count: 1 + +Tag #42 [region] + tagno: 1124 (Payloadformat) + type: 6 (string) + offset: 2209 + count: 1 + +Tag #43 [region] + tagno: 1126 (Payloadflags) + type: 6 (string) + offset: 2214 + count: 1 + +Tag #44 [region] + tagno: 1132 (Platform) + type: 6 (string) + offset: 2215 + count: 1 + +Tag #45 [region] + tagno: 1140 (Filecolors) + type: 4 (int32) + offset: 2228 + count: 20 + +Tag #46 [region] + tagno: 1141 (Fileclass) + type: 4 (int32) + offset: 2308 + count: 20 + +Tag #47 [region] + tagno: 1142 (Classdict) + type: 8 (argv) + offset: 2388 + count: 2 + +Tag #48 [region] + tagno: 5011 (Filedigestalgo) + type: 4 (int32) + offset: 2412 + count: 1 + +Tag #49 [region] + tagno: 5062 (Encoding) + type: 6 (string) + offset: 2416 + count: 1 + +Tag #50 [region] + tagno: 5092 (Payloaddigest) + type: 8 (argv) + offset: 2422 + count: 1 + +Tag #51 [region] + tagno: 5093 (Payloaddigestalgo) + type: 4 (int32) + offset: 2488 + count: 1 + +Tag #52 [region] + tagno: 5097 (Payloaddigestalt) + type: 8 (argv) + offset: 2492 + count: 1 + diff --git a/tests/pinned/rpmsigdig6.sh b/tests/pinned/rpmsigdig6.sh new file mode 100755 index 0000000000..ad30089048 --- /dev/null +++ b/tests/pinned/rpmsigdig6.sh @@ -0,0 +1,5 @@ +ver=6 +pkg=/build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm + +source /pinned/common/buildrepr.sh +source /pinned/common/rpmsigdig.sh diff --git a/tests/pinned/rpmsigdig6.txt b/tests/pinned/rpmsigdig6.txt new file mode 100644 index 0000000000..c58e1239ce --- /dev/null +++ b/tests/pinned/rpmsigdig6.txt @@ -0,0 +1,11 @@ +SHA256HEADER: 8ab715bef152a18bef62df29805b9d1b8f2c5d92955a1a7fa2491967cc17c035 +SHA1HEADER: 7a8ad9bdb9b0910fa54724f7bdac8ac49d992d2f +SIGMD5: 87da99d1993240e2d698f5b3dbcbd44c +PAYLOADDIGEST: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 +PAYLOADDIGESTALT: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 +/build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm: + Header SHA256 digest: OK + Header SHA1 digest: OK + Payload SHA256 ALT digest: OK + Payload SHA256 digest: OK + MD5 digest: OK diff --git a/tests/rpmpkgfmt.at b/tests/rpmpkgfmt.at index f42182b92d..bb4d824139 100644 --- a/tests/rpmpkgfmt.at +++ b/tests/rpmpkgfmt.at @@ -14,3 +14,20 @@ runroot_other ${RPM_CONFIGDIR_PATH}/rpmdump ${pkg} [expout], []) RPMTEST_CLEANUP + +AT_SETUP([rpm v6 format]) +AT_KEYWORDS([pkgformat v6]) +RPMDB_INIT + +RPMTEST_CHECK([ +cp /data/misc/rpmdump6.txt expout +ver=6 +pkg=/build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm +source /pinned/common/buildrepr.sh + +runroot_other ${RPM_CONFIGDIR_PATH}/rpmdump ${pkg} +], +[0], +[expout], +[]) +RPMTEST_CLEANUP diff --git a/tests/rpmsigdig.at b/tests/rpmsigdig.at index 5e455b8ce6..5f9d3d0ed8 100644 --- a/tests/rpmsigdig.at +++ b/tests/rpmsigdig.at @@ -155,9 +155,10 @@ RPMTEST_CLEANUP # ------------------------------ # Reproducably build and verify a package AT_SETUP([rpmkeys -Kv 2]) -AT_KEYWORDS([rpmkeys digest]) +AT_KEYWORDS([rpmkeys digest v4 v6]) RPMDB_INIT RPMTEST_CHECK_PINNED([rpmsigdig]) +RPMTEST_CHECK_PINNED([rpmsigdig6]) RPMTEST_CLEANUP # ------------------------------ From a381ff42f8e6832effb918d827d88004ecbd4c49 Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Mon, 19 Feb 2024 11:39:21 +0200 Subject: [PATCH 3/9] First commit of rpm v6: add version switch, drop obsolete signature tags Add %_rpmfilever macro to control which format we create, with experimental disclaimer: the v6 spec hasn't been finalized yet, this is merely the beginning of the implementation for the parts we know already: Obsolete crypto (MD5 and SHA1) tags and the unreliable size tags in the signature header are dropped. --- build/pack.c | 21 ++++++++++----- build/rpmbuild_internal.h | 1 + build/spec.c | 1 + lib/signature.c | 8 +++++- lib/signature.h | 4 ++- macros.in | 6 +++++ tests/data/misc/rpmdump6.txt | 44 ++++++++------------------------ tests/pinned/common/buildrepr.sh | 1 + tests/pinned/rpmsigdig6.txt | 8 +++--- 9 files changed, 46 insertions(+), 48 deletions(-) diff --git a/build/pack.c b/build/pack.c index 3914704ab3..7d818e3c98 100644 --- a/build/pack.c +++ b/build/pack.c @@ -504,10 +504,12 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, sigStart = Ftell(fd); /* Generate and write a placeholder signature header */ - SHA1 = nullDigest(RPM_HASH_SHA1, 1); + if (pkg->rpmver < 6) { + SHA1 = nullDigest(RPM_HASH_SHA1, 1); + MD5 = nullDigest(RPM_HASH_MD5, 0); + } SHA256 = nullDigest(RPM_HASH_SHA256, 1); - MD5 = nullDigest(RPM_HASH_MD5, 0); - if (rpmGenerateSignature(SHA256, SHA1, MD5, 0, 0, fd)) + if (rpmGenerateSignature(SHA256, SHA1, MD5, 0, 0, fd, pkg->rpmver)) goto exit; SHA1 = _free(SHA1); SHA256 = _free(SHA256); @@ -543,9 +545,12 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, if (writeHdr(fd, pkg->header)) goto exit; - /* Calculate digests: SHA on header, legacy MD5 on header + payload */ - fdInitDigestID(fd, RPM_HASH_MD5, RPMTAG_SIGMD5, 0); - fdInitDigestID(fd, RPM_HASH_SHA1, RPMTAG_SHA1HEADER, 0); + /* Calculate the digests */ + if (pkg->rpmver < 6) { + /* SHA1 and legacy MD5 on header + payload only in v4 */ + fdInitDigestID(fd, RPM_HASH_MD5, RPMTAG_SIGMD5, 0); + fdInitDigestID(fd, RPM_HASH_SHA1, RPMTAG_SHA1HEADER, 0); + } fdInitDigestID(fd, RPM_HASH_SHA256, RPMTAG_SHA256HEADER, 0); if (fdConsume(fd, hdrStart, payloadStart - hdrStart)) goto exit; @@ -560,8 +565,10 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, goto exit; /* Generate the signature. Now with right values */ - if (rpmGenerateSignature(SHA256, SHA1, MD5, payloadEnd - hdrStart, archiveSize, fd)) + if (rpmGenerateSignature(SHA256, SHA1, MD5, payloadEnd - hdrStart, + archiveSize, fd, pkg->rpmver)) { goto exit; + } rc = RPMRC_OK; diff --git a/build/rpmbuild_internal.h b/build/rpmbuild_internal.h index 0785917315..715b222790 100644 --- a/build/rpmbuild_internal.h +++ b/build/rpmbuild_internal.h @@ -176,6 +176,7 @@ struct Package_s { rpmds dependencies[PACKAGE_NUM_DEPS]; rpmfiles cpioList; ARGV_t dpaths; + int rpmver; /* v4, v6? */ struct Source * icon; diff --git a/build/spec.c b/build/spec.c index 21cf8d1544..c8485e15e8 100644 --- a/build/spec.c +++ b/build/spec.c @@ -113,6 +113,7 @@ Package newPackage(const char *name, rpmstrPool pool, Package *pkglist) p->fileRenameMap = NULL; p->pool = rpmstrPoolLink(pool); p->dpaths = NULL; + p->rpmver = rpmExpandNumeric("%_rpmfilever"); if (name) p->name = rpmstrPoolId(p->pool, name, 1); diff --git a/lib/signature.c b/lib/signature.c index 330ffb23f0..0ad4c63685 100644 --- a/lib/signature.c +++ b/lib/signature.c @@ -110,7 +110,8 @@ int rpmWriteSignature(FD_t fd, Header sigh) } rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, - rpm_loff_t size, rpm_loff_t payloadSize, FD_t fd) + rpm_loff_t size, rpm_loff_t payloadSize, FD_t fd, + int rpmver) { Header sig = headerNew(); struct rpmtd_s td; @@ -131,6 +132,10 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, headerPut(sig, &td, HEADERPUT_DEFAULT); } + /* Skip all the old stuff v6 doesn't have */ + if (rpmver >= 6) + goto reserve; + if (SHA1) { rpmtdReset(&td); td.tag = RPMSIGTAG_SHA1; @@ -193,6 +198,7 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, spaceSize -= newsigSize - oldsigSize; } +reserve: if (gpgSize > 0) spaceSize += gpgSize; diff --git a/lib/signature.h b/lib/signature.h index 56d07f5961..33eec6d4d1 100644 --- a/lib/signature.h +++ b/lib/signature.h @@ -37,9 +37,11 @@ int rpmWriteSignature(FD_t fd, Header h); * @param size size of header * @param payloadSize size of archive * @param fd output file + * @param rpmver rpm format version (4 or 6) */ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, - rpm_loff_t size, rpm_loff_t payloadSize, FD_t fd); + rpm_loff_t size, rpm_loff_t payloadSize, FD_t fd, + int rpmver); #ifdef __cplusplus } diff --git a/macros.in b/macros.in index 3cdd45d76a..39e7e766b9 100644 --- a/macros.in +++ b/macros.in @@ -255,6 +255,12 @@ Supplements: (%{name} = %{version}-%{release} and langpacks-%{1})\ # (legacy). %_rpmfilename %{_build_name_fmt} +# EXPERIMENTAL +# This does NOT generate actual v6 format, the exact format +# hasn't been finalized yet. +# Which rpm format go generate (4 or 6) +%_rpmfilever 4 + # The directory where sources/patches from a source package will be # installed. This is also where sources/patches are found when building. %_sourcedir %{_topdir}/SOURCES diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index eeac859b21..32c6742d8a 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -8,60 +8,36 @@ Sigtype: 5 Signature: Header magic: 1e8ad8e (reserved: 0) -Index entries: 7 (112 bytes) -Data size: 4276 bytes -Header size: 4388 bytes -Padding: 4 bytes -Region entries 7 -Region size 128 +Index entries: 3 (48 bytes) +Data size: 4209 bytes +Header size: 4257 bytes +Padding: 7 bytes +Region entries 3 +Region size 64 Dribbles: 0 Tag #0 [region] tagno: 62 (Headersignatures) type: 7 (blob) - offset: 4260 + offset: 4193 count: 16 region trailer tagno: 62 (Headersignatures) type: 7 (blob) - offset: -112 + offset: -48 count: 16 Tag #1 [region] - tagno: 269 (Sha1) + tagno: 273 (Sha256) type: 6 (string) offset: 0 count: 1 Tag #2 [region] - tagno: 273 (Sha256) - type: 6 (string) - offset: 41 - count: 1 - -Tag #3 [region] - tagno: 1000 (Size) - type: 4 (int32) - offset: 108 - count: 1 - -Tag #4 [region] - tagno: 1004 (Md5) - type: 7 (blob) - offset: 112 - count: 16 - -Tag #5 [region] - tagno: 1007 (Payloadsize) - type: 4 (int32) - offset: 128 - count: 1 - -Tag #6 [region] tagno: 1008 (Reservedspace) type: 7 (blob) - offset: 132 + offset: 65 count: 4128 Header: diff --git a/tests/pinned/common/buildrepr.sh b/tests/pinned/common/buildrepr.sh index e4bacb2db3..a4a88b26ac 100644 --- a/tests/pinned/common/buildrepr.sh +++ b/tests/pinned/common/buildrepr.sh @@ -1,4 +1,5 @@ runroot rpmbuild -bb --quiet \ + --define "_rpmfilever ${ver}" \ --define "_rpmdir %{_topdir}/RPMS/${ver}" \ --define "optflags -O2 -g" \ --define "_target_platform noarch-linux" \ diff --git a/tests/pinned/rpmsigdig6.txt b/tests/pinned/rpmsigdig6.txt index c58e1239ce..b91ca3fa5c 100644 --- a/tests/pinned/rpmsigdig6.txt +++ b/tests/pinned/rpmsigdig6.txt @@ -1,11 +1,9 @@ -SHA256HEADER: 8ab715bef152a18bef62df29805b9d1b8f2c5d92955a1a7fa2491967cc17c035 -SHA1HEADER: 7a8ad9bdb9b0910fa54724f7bdac8ac49d992d2f -SIGMD5: 87da99d1993240e2d698f5b3dbcbd44c +SHA256HEADER: 56435bfb3c842ac7bcd30d2955607624bf7035a10555ea0efd74a4eaa8216447 +SHA1HEADER: (none) +SIGMD5: (none) PAYLOADDIGEST: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 PAYLOADDIGESTALT: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 /build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm: Header SHA256 digest: OK - Header SHA1 digest: OK Payload SHA256 ALT digest: OK Payload SHA256 digest: OK - MD5 digest: OK From 784bb9076d614da33d29123f5ef6236a57d38463 Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 10:26:22 +0200 Subject: [PATCH 4/9] Introduce new payloadsize + alt tags in the v6 main header The multitude of size tags in the v4 signature header are effectively useless because they're unsigned and so entirely untrustworthy. Having a reliable size available seems like a potentially useful thing to have, so lets yet another set of size tags, but this time in the main header like the payload digests. And like with the digests, have one for the compressed and alt-version for the uncompressed size. Unlike the payload digests, this is only in v6 to avoid introducing what is technically an v4 incompatibility due the int64 type. While this is neither a signature or digest, having the size tested next to the digests seems useful. The reproducability test doesn't use compression so both sizes are equal. --- build/pack.c | 25 +++++++++++++++++++------ include/rpm/rpmtag.h | 2 ++ tests/data/misc/rpmdump6.txt | 26 +++++++++++++++++++------- tests/pinned/common/rpmsigdig.sh | 2 +- tests/pinned/rpmsigdig.txt | 2 ++ tests/pinned/rpmsigdig6.txt | 4 +++- tests/rpmgeneral.at | 2 ++ 7 files changed, 48 insertions(+), 15 deletions(-) diff --git a/build/pack.c b/build/pack.c index 7d818e3c98..7bf305b34b 100644 --- a/build/pack.c +++ b/build/pack.c @@ -457,7 +457,8 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, char * upld = NULL; uint32_t pld_algo = RPM_HASH_SHA256; /* TODO: macro configuration */ rpmRC rc = RPMRC_FAIL; /* assume failure */ - rpm_loff_t archiveSize = 0; + rpm_loff_t archiveSize = 0; /* uncompressed */ + rpm_loff_t payloadSize = 0; /* compressed */ off_t sigStart, hdrStart, payloadStart, payloadEnd; if (pkgidp) @@ -475,13 +476,17 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, headerPutString(pkg->header, RPMTAG_COOKIE, *cookie); } - /* Create a dummy payload digests to get the header size right */ + /* Create a dummy payload digests + size to get the header size right */ pld = nullDigest(pld_algo, 1); headerPutUint32(pkg->header, RPMTAG_PAYLOADDIGESTALGO, &pld_algo, 1); headerPutString(pkg->header, RPMTAG_PAYLOADDIGEST, pld); headerPutString(pkg->header, RPMTAG_PAYLOADDIGESTALT, pld); pld = _free(pld); - + if (pkg->rpmver >= 6) { + headerPutUint64(pkg->header, RPMTAG_PAYLOADSIZE, &payloadSize, 1); + headerPutUint64(pkg->header, RPMTAG_PAYLOADSIZEALT, &archiveSize, 1); + } + /* Check for UTF-8 encoding of string tags, add encoding tag if all good */ if (checkForEncoding(pkg->header, 1)) goto exit; @@ -525,20 +530,28 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, if (cpio_doio(fd, pkg, rpmio_flags, pld_algo, &archiveSize, &upld)) goto exit; payloadEnd = Ftell(fd); + payloadSize = payloadEnd - payloadStart; /* Re-read payload to calculate compressed digest */ fdInitDigestID(fd, pld_algo, RPMTAG_PAYLOADDIGEST, 0); - if (fdConsume(fd, payloadStart, payloadEnd - payloadStart)) + if (fdConsume(fd, payloadStart, payloadSize)) goto exit; fdFiniDigest(fd, RPMTAG_PAYLOADDIGEST, (void **)&pld, NULL, 1); - /* Insert the payload digests in main header */ + /* Insert the payload digests + size in main header */ headerDel(pkg->header, RPMTAG_PAYLOADDIGEST); headerPutString(pkg->header, RPMTAG_PAYLOADDIGEST, pld); headerDel(pkg->header, RPMTAG_PAYLOADDIGESTALT); headerPutString(pkg->header, RPMTAG_PAYLOADDIGESTALT, upld); pld = _free(pld); + if (pkg->rpmver >= 6) { + headerDel(pkg->header, RPMTAG_PAYLOADSIZE); + headerPutUint64(pkg->header, RPMTAG_PAYLOADSIZE, &payloadSize, 1); + headerDel(pkg->header, RPMTAG_PAYLOADSIZEALT); + headerPutUint64(pkg->header, RPMTAG_PAYLOADSIZEALT, &archiveSize, 1); + } + /* Write the final header */ if (fdJump(fd, hdrStart)) goto exit; @@ -557,7 +570,7 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, fdFiniDigest(fd, RPMTAG_SHA1HEADER, (void **)&SHA1, NULL, 1); fdFiniDigest(fd, RPMTAG_SHA256HEADER, (void **)&SHA256, NULL, 1); - if (fdConsume(fd, 0, payloadEnd - payloadStart)) + if (fdConsume(fd, 0, payloadSize)) goto exit; fdFiniDigest(fd, RPMTAG_SIGMD5, (void **)&MD5, NULL, 0); diff --git a/include/rpm/rpmtag.h b/include/rpm/rpmtag.h index 3af1bd92c8..24a8a38b92 100644 --- a/include/rpm/rpmtag.h +++ b/include/rpm/rpmtag.h @@ -388,6 +388,8 @@ typedef enum rpmTag_e { RPMTAG_SYSUSERS = 5109, /* s[] extension */ RPMTAG_BUILDSYSTEM = 5110, /* internal */ RPMTAG_BUILDOPTION = 5111, /* internal */ + RPMTAG_PAYLOADSIZE = 5112, /* l */ + RPMTAG_PAYLOADSIZEALT = 5113, /* l */ RPMTAG_FIRSTFREE_TAG /*!< internal */ } rpmTag; diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index 32c6742d8a..66406e39ed 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -42,24 +42,24 @@ Tag #2 [region] Header: Header magic: 1e8ad8e (reserved: 0) -Index entries: 53 (848 bytes) -Data size: 2573 bytes -Header size: 3421 bytes +Index entries: 55 (880 bytes) +Data size: 2592 bytes +Header size: 3472 bytes -Region entries 53 -Region size 864 +Region entries 55 +Region size 896 Dribbles: 0 Tag #0 [region] tagno: 63 (Headerimmutable) type: 7 (blob) - offset: 2557 + offset: 2576 count: 16 region trailer tagno: 63 (Headerimmutable) type: 7 (blob) - offset: -848 + offset: -880 count: 16 Tag #1 [region] @@ -374,3 +374,15 @@ Tag #52 [region] offset: 2492 count: 1 +Tag #53 [region] + tagno: 5112 (Payloadsize) + type: 5 (int64) + offset: 2560 + count: 1 + +Tag #54 [region] + tagno: 5113 (Payloadsizealt) + type: 5 (int64) + offset: 2568 + count: 1 + diff --git a/tests/pinned/common/rpmsigdig.sh b/tests/pinned/common/rpmsigdig.sh index 120ec4af1a..c0905bfaf5 100644 --- a/tests/pinned/common/rpmsigdig.sh +++ b/tests/pinned/common/rpmsigdig.sh @@ -1,5 +1,5 @@ -for v in SHA256HEADER SHA1HEADER SIGMD5 PAYLOADDIGEST PAYLOADDIGESTALT; do +for v in SHA256HEADER SHA1HEADER SIGMD5 PAYLOADDIGEST PAYLOADDIGESTALT PAYLOADSIZE PAYLOADSIZEALT; do runroot rpm -q --qf "${v}: %{${v}}\n" ${pkg} done runroot rpmkeys -Kv ${pkg} diff --git a/tests/pinned/rpmsigdig.txt b/tests/pinned/rpmsigdig.txt index dd36927654..37d263d777 100644 --- a/tests/pinned/rpmsigdig.txt +++ b/tests/pinned/rpmsigdig.txt @@ -3,6 +3,8 @@ SHA1HEADER: ead61996e3f1588d5c47b844ed75ec46079ceaf5 SIGMD5: a7039f3a5b1aad754ff0022694f8c2f3 PAYLOADDIGEST: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 PAYLOADDIGESTALT: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 +PAYLOADSIZE: (none) +PAYLOADSIZEALT: (none) /build/RPMS/4/noarch/attrtest-1.0-1.noarch.rpm: Header SHA256 digest: OK Header SHA1 digest: OK diff --git a/tests/pinned/rpmsigdig6.txt b/tests/pinned/rpmsigdig6.txt index b91ca3fa5c..9f37b6b191 100644 --- a/tests/pinned/rpmsigdig6.txt +++ b/tests/pinned/rpmsigdig6.txt @@ -1,8 +1,10 @@ -SHA256HEADER: 56435bfb3c842ac7bcd30d2955607624bf7035a10555ea0efd74a4eaa8216447 +SHA256HEADER: 3e3247383a6c2958d0cc308bdcae21673097ba097b2dacea1eb89f3a525363f8 SHA1HEADER: (none) SIGMD5: (none) PAYLOADDIGEST: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 PAYLOADDIGESTALT: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 +PAYLOADSIZE: 2684 +PAYLOADSIZEALT: 2684 /build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm: Header SHA256 digest: OK Payload SHA256 ALT digest: OK diff --git a/tests/rpmgeneral.at b/tests/rpmgeneral.at index 7378b6a7a8..8616a7d98f 100644 --- a/tests/rpmgeneral.at +++ b/tests/rpmgeneral.at @@ -209,6 +209,8 @@ PAYLOADDIGESTALGO PAYLOADDIGESTALT PAYLOADFLAGS PAYLOADFORMAT +PAYLOADSIZE +PAYLOADSIZEALT PKGID PLATFORM POLICIES From 87ac9f94eea2181af404bb9053fabf6d84ec03da Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 10:45:33 +0200 Subject: [PATCH 5/9] Always use long filesizes and the "new" non-cpio payload format on v6 Switching to the long tags implicitly changes to the "new" payload format which in v4 was used only for packages with >= 4GB files. It's noteworthy how much smaller the payload is: the new format has much less overhead due to not carrying duplicate metadata. Also noteworthy is that we still cannot change the name of the payload format stored in the header, as doing so would cause a gratuitous incompatibility with rpm >= 4.12 which can handle the payload format just fine. So rpmlib(LargeFiles) dependency remains the means to track the actual payload - as an acute observer will note, the number of dependencies in the dump grows by one. --- build/files.c | 7 +- tests/data/misc/rpmdump6.txt | 196 +++++++++++++++++------------------ tests/pinned/rpmsigdig6.txt | 10 +- 3 files changed, 107 insertions(+), 106 deletions(-) diff --git a/build/files.c b/build/files.c index 67bde09753..ed6a9b56b4 100644 --- a/build/files.c +++ b/build/files.c @@ -1176,8 +1176,8 @@ static void genCpioListAndHeader(FileList fl, Package pkg, int isSrc) headerPutString(h, RPMTAG_FILEGROUPNAME, rpmstrPoolStr(fl->pool, flp->gname)); - /* Only use 64bit filesizes tag if required. */ - if (fl->largeFiles) { + /* Use 64bit filesizes always on v6, on older only if required. */ + if (pkg->rpmver >= 6 || fl->largeFiles) { rpm_loff_t rsize64 = (rpm_loff_t)flp->fl_size; headerPutUint64(h, RPMTAG_LONGFILESIZES, &rsize64, 1); (void) rpmlibNeedsFeature(pkg, "LargeFiles", "4.12.0-1"); @@ -1280,7 +1280,8 @@ static void genCpioListAndHeader(FileList fl, Package pkg, int isSrc) } pkg->dpaths[npaths] = NULL; - if (totalFileSize < UINT32_MAX) { + /* Use 64bit sizes always on v6, on older only if required. */ + if (pkg->rpmver < 6 && totalFileSize < UINT32_MAX) { rpm_off_t totalsize = totalFileSize; headerPutUint32(h, RPMTAG_SIZE, &totalsize, 1); } else { diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index 66406e39ed..c0884a170f 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -43,8 +43,8 @@ Tag #2 [region] Header: Header magic: 1e8ad8e (reserved: 0) Index entries: 55 (880 bytes) -Data size: 2592 bytes -Header size: 3472 bytes +Data size: 2712 bytes +Header size: 3592 bytes Region entries 55 Region size 896 @@ -53,7 +53,7 @@ Dribbles: 0 Tag #0 [region] tagno: 63 (Headerimmutable) type: 7 (blob) - offset: 2576 + offset: 2696 count: 16 region trailer @@ -111,278 +111,278 @@ Tag #8 [region] count: 1 Tag #9 [region] - tagno: 1009 (Size) - type: 4 (int32) - offset: 88 - count: 1 - -Tag #10 [region] tagno: 1014 (License) type: 6 (string) - offset: 92 + offset: 85 count: 1 -Tag #11 [region] +Tag #10 [region] tagno: 1016 (Group) type: 9 (i18nstring) - offset: 96 + offset: 89 count: 1 -Tag #12 [region] +Tag #11 [region] tagno: 1021 (Os) type: 6 (string) - offset: 104 + offset: 97 count: 1 -Tag #13 [region] +Tag #12 [region] tagno: 1022 (Arch) type: 6 (string) - offset: 110 + offset: 103 count: 1 -Tag #14 [region] - tagno: 1028 (Filesizes) - type: 4 (int32) - offset: 120 - count: 20 - -Tag #15 [region] +Tag #13 [region] tagno: 1030 (Filemodes) type: 3 (int16) - offset: 200 + offset: 110 count: 20 -Tag #16 [region] +Tag #14 [region] tagno: 1033 (Filerdevs) type: 3 (int16) - offset: 240 + offset: 150 count: 20 -Tag #17 [region] +Tag #15 [region] tagno: 1034 (Filemtimes) type: 4 (int32) - offset: 280 + offset: 192 count: 20 -Tag #18 [region] +Tag #16 [region] tagno: 1035 (Filedigests) type: 8 (argv) - offset: 360 + offset: 272 count: 20 -Tag #19 [region] +Tag #17 [region] tagno: 1036 (Filelinktos) type: 8 (argv) - offset: 1020 + offset: 932 count: 20 -Tag #20 [region] +Tag #18 [region] tagno: 1037 (Fileflags) type: 4 (int32) - offset: 1040 + offset: 952 count: 20 -Tag #21 [region] +Tag #19 [region] tagno: 1039 (Fileusername) type: 8 (argv) - offset: 1120 + offset: 1032 count: 20 -Tag #22 [region] +Tag #20 [region] tagno: 1040 (Filegroupname) type: 8 (argv) - offset: 1218 + offset: 1130 count: 20 -Tag #23 [region] +Tag #21 [region] tagno: 1044 (Sourcerpm) type: 6 (string) - offset: 1307 + offset: 1219 count: 1 -Tag #24 [region] +Tag #22 [region] tagno: 1045 (Fileverifyflags) type: 4 (int32) - offset: 1332 + offset: 1244 count: 20 -Tag #25 [region] +Tag #23 [region] tagno: 1047 (Providename) type: 8 (argv) - offset: 1412 + offset: 1324 count: 1 -Tag #26 [region] +Tag #24 [region] tagno: 1048 (Requireflags) type: 4 (int32) - offset: 1424 - count: 15 + offset: 1336 + count: 16 -Tag #27 [region] +Tag #25 [region] tagno: 1049 (Requirename) type: 8 (argv) - offset: 1484 - count: 15 + offset: 1400 + count: 16 -Tag #28 [region] +Tag #26 [region] tagno: 1050 (Requireversion) type: 8 (argv) - offset: 1697 - count: 15 + offset: 1632 + count: 16 -Tag #29 [region] +Tag #27 [region] tagno: 1064 (Rpmversion) type: 6 (string) - offset: 1731 + offset: 1675 count: 1 -Tag #30 [region] +Tag #28 [region] tagno: 1080 (Changelogtime) type: 4 (int32) - offset: 1740 + offset: 1684 count: 1 -Tag #31 [region] +Tag #29 [region] tagno: 1081 (Changelogname) type: 8 (argv) - offset: 1744 + offset: 1688 count: 1 -Tag #32 [region] +Tag #30 [region] tagno: 1082 (Changelogtext) type: 8 (argv) - offset: 1782 + offset: 1726 count: 1 -Tag #33 [region] +Tag #31 [region] tagno: 1095 (Filedevices) type: 4 (int32) - offset: 1800 + offset: 1744 count: 20 -Tag #34 [region] +Tag #32 [region] tagno: 1096 (Fileinodes) type: 4 (int32) - offset: 1880 + offset: 1824 count: 20 -Tag #35 [region] +Tag #33 [region] tagno: 1097 (Filelangs) type: 8 (argv) - offset: 1960 + offset: 1904 count: 20 -Tag #36 [region] +Tag #34 [region] tagno: 1112 (Provideflags) type: 4 (int32) - offset: 1980 + offset: 1924 count: 1 -Tag #37 [region] +Tag #35 [region] tagno: 1113 (Provideversion) type: 8 (argv) - offset: 1984 + offset: 1928 count: 1 -Tag #38 [region] +Tag #36 [region] tagno: 1116 (Dirindexes) type: 4 (int32) - offset: 1992 + offset: 1936 count: 20 -Tag #39 [region] +Tag #37 [region] tagno: 1117 (Basenames) type: 8 (argv) - offset: 2072 + offset: 2016 count: 20 -Tag #40 [region] +Tag #38 [region] tagno: 1118 (Dirnames) type: 8 (argv) - offset: 2162 + offset: 2106 count: 10 -Tag #41 [region] +Tag #39 [region] tagno: 1122 (Optflags) type: 6 (string) - offset: 2202 + offset: 2146 count: 1 -Tag #42 [region] +Tag #40 [region] tagno: 1124 (Payloadformat) type: 6 (string) - offset: 2209 + offset: 2153 count: 1 -Tag #43 [region] +Tag #41 [region] tagno: 1126 (Payloadflags) type: 6 (string) - offset: 2214 + offset: 2158 count: 1 -Tag #44 [region] +Tag #42 [region] tagno: 1132 (Platform) type: 6 (string) - offset: 2215 + offset: 2159 count: 1 -Tag #45 [region] +Tag #43 [region] tagno: 1140 (Filecolors) type: 4 (int32) - offset: 2228 + offset: 2172 count: 20 -Tag #46 [region] +Tag #44 [region] tagno: 1141 (Fileclass) type: 4 (int32) - offset: 2308 + offset: 2252 count: 20 -Tag #47 [region] +Tag #45 [region] tagno: 1142 (Classdict) type: 8 (argv) - offset: 2388 + offset: 2332 count: 2 +Tag #46 [region] + tagno: 5008 (Longfilesizes) + type: 5 (int64) + offset: 2360 + count: 20 + +Tag #47 [region] + tagno: 5009 (Longsize) + type: 5 (int64) + offset: 2520 + count: 1 + Tag #48 [region] tagno: 5011 (Filedigestalgo) type: 4 (int32) - offset: 2412 + offset: 2528 count: 1 Tag #49 [region] tagno: 5062 (Encoding) type: 6 (string) - offset: 2416 + offset: 2532 count: 1 Tag #50 [region] tagno: 5092 (Payloaddigest) type: 8 (argv) - offset: 2422 + offset: 2538 count: 1 Tag #51 [region] tagno: 5093 (Payloaddigestalgo) type: 4 (int32) - offset: 2488 + offset: 2604 count: 1 Tag #52 [region] tagno: 5097 (Payloaddigestalt) type: 8 (argv) - offset: 2492 + offset: 2608 count: 1 Tag #53 [region] tagno: 5112 (Payloadsize) type: 5 (int64) - offset: 2560 + offset: 2680 count: 1 Tag #54 [region] tagno: 5113 (Payloadsizealt) type: 5 (int64) - offset: 2568 + offset: 2688 count: 1 diff --git a/tests/pinned/rpmsigdig6.txt b/tests/pinned/rpmsigdig6.txt index 9f37b6b191..898b9187a8 100644 --- a/tests/pinned/rpmsigdig6.txt +++ b/tests/pinned/rpmsigdig6.txt @@ -1,10 +1,10 @@ -SHA256HEADER: 3e3247383a6c2958d0cc308bdcae21673097ba097b2dacea1eb89f3a525363f8 +SHA256HEADER: 37992f97f4116f6969eeffdc4dc41638630c5dedaa8ef0f466650aea326ec84a SHA1HEADER: (none) SIGMD5: (none) -PAYLOADDIGEST: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 -PAYLOADDIGESTALT: 116ce41ebb72f1877cda3d7dedaf5b78770e202d6389ade4e415d78548d703a8 -PAYLOADSIZE: 2684 -PAYLOADSIZEALT: 2684 +PAYLOADDIGEST: 2f916d301d47ee34f16b74c7c49d11448fceb5487c6ba53c04950423fffeab95 +PAYLOADDIGESTALT: 2f916d301d47ee34f16b74c7c49d11448fceb5487c6ba53c04950423fffeab95 +PAYLOADSIZE: 604 +PAYLOADSIZEALT: 604 /build/RPMS/6/noarch/attrtest-1.0-1.noarch.rpm: Header SHA256 digest: OK Payload SHA256 ALT digest: OK From 55d928824a25480bb77d22cb7a1fad67f11a537a Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 11:54:17 +0200 Subject: [PATCH 6/9] Add a new unique v6-only reserved signature tag below header base Unfortunately RPMSIGTAG_RESERVEDSPACE clashes with RPMTAG_INSTALLTIME so we can't use it in v6. Introduce a new tag at the top of the signature range to make sure its always the last one - on v6 packages that is. For v4 packages we're better off not messing with the existing tag at this point, we just need to handle both. --- include/rpm/rpmtag.h | 3 +++ lib/signature.c | 9 ++++++--- sign/rpmgensig.c | 7 ++++++- tests/data/misc/rpmdump6.txt | 2 +- tools/rpmdump.c | 1 + 5 files changed, 17 insertions(+), 5 deletions(-) diff --git a/include/rpm/rpmtag.h b/include/rpm/rpmtag.h index 24a8a38b92..2ac0b61c27 100644 --- a/include/rpm/rpmtag.h +++ b/include/rpm/rpmtag.h @@ -23,6 +23,7 @@ extern "C" { #define HEADER_REGIONS 64 #define HEADER_I18NTABLE 100 #define HEADER_SIGBASE 256 +#define HEADER_SIGTOP 999 #define HEADER_TAGBASE 1000 /** \ingroup rpmtag @@ -67,6 +68,7 @@ typedef enum rpmTag_e { /* RPMTAG_SIG_BASE+19 reserved for RPMSIGTAG_FILESIGNATURELENGTH */ RPMTAG_VERITYSIGNATURES = RPMTAG_SIG_BASE+20, /* s[] */ RPMTAG_VERITYSIGNATUREALGO = RPMTAG_SIG_BASE+21, /* i */ + RPMTAG_SIG_TOP = HEADER_SIGTOP, RPMTAG_NAME = 1000, /* s */ #define RPMTAG_N RPMTAG_NAME /* s */ @@ -449,6 +451,7 @@ typedef enum rpmSigTag_e { RPMSIGTAG_FILESIGNATURELENGTH = RPMTAG_SIG_BASE + 19, RPMSIGTAG_VERITYSIGNATURES = RPMTAG_VERITYSIGNATURES, RPMSIGTAG_VERITYSIGNATUREALGO = RPMTAG_VERITYSIGNATUREALGO, + RPMSIGTAG_RESERVED = RPMTAG_SIG_TOP, } rpmSigTag; diff --git a/lib/signature.c b/lib/signature.c index 0ad4c63685..6a1455b131 100644 --- a/lib/signature.c +++ b/lib/signature.c @@ -121,6 +121,7 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, int gpgSize = rpmExpandNumeric("%{__gpg_reserved_space}"); rpm_off_t size32 = size; rpm_off_t payloadSize32 = payloadSize; + rpmTagVal reserveTag = RPMSIGTAG_RESERVEDSPACE; /* Prepare signature */ if (SHA256) { @@ -133,8 +134,10 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, } /* Skip all the old stuff v6 doesn't have */ - if (rpmver >= 6) + if (rpmver >= 6) { + reserveTag = RPMSIGTAG_RESERVED; goto reserve; + } if (SHA1) { rpmtdReset(&td); @@ -171,7 +174,7 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, * Put the 64bit size variants into the header, but * modify spaceSize so that the resulting header has * the same size. Note that this only works if all tags - * with a lower number than RPMSIGTAG_RESERVEDSPACE are + * with a lower number than RPMSIGTAG_RESERVEDSPACE/PAD are * already added and no tag with a higher number is * added yet. */ @@ -205,7 +208,7 @@ rpmRC rpmGenerateSignature(char *SHA256, char *SHA1, uint8_t *MD5, if (spaceSize > 0) { reservedSpace = xcalloc(spaceSize, sizeof(char)); rpmtdReset(&td); - td.tag = RPMSIGTAG_RESERVEDSPACE; + td.tag = reserveTag; td.count = spaceSize; td.type = RPM_BIN_TYPE; td.data = reservedSpace; diff --git a/sign/rpmgensig.c b/sign/rpmgensig.c index f82a596e6c..28161e77d2 100644 --- a/sign/rpmgensig.c +++ b/sign/rpmgensig.c @@ -565,6 +565,7 @@ static int rpmSign(const char *rpm, int deleting, int flags) struct sigTarget_s sigt_v4; unsigned int origSigSize; int insSig = 0; + rpmTagVal reserveTag = RPMSIGTAG_RESERVEDSPACE; fprintf(stdout, "%s:\n", rpm); @@ -650,8 +651,12 @@ static int rpmSign(const char *rpm, int deleting, int flags) res = -1; } + /* Only v6 packages have this */ + if (headerIsEntry(h, RPMSIGTAG_RESERVED)) + reserveTag = RPMSIGTAG_RESERVED; + /* Adjust reserved size for added/removed signatures */ - if (headerGet(sigh, RPMSIGTAG_RESERVEDSPACE, &utd, HEADERGET_MINMEM)) { + if (headerGet(sigh, reserveTag, &utd, HEADERGET_MINMEM)) { int diff = headerSizeof(sigh, HEADER_MAGIC_YES) - origSigSize; /* diff can be zero if nothing was added or removed */ diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index c0884a170f..61e2d92a26 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -35,7 +35,7 @@ Tag #1 [region] count: 1 Tag #2 [region] - tagno: 1008 (Reservedspace) + tagno: 999 (Reserved) type: 7 (blob) offset: 65 count: 4128 diff --git a/tools/rpmdump.c b/tools/rpmdump.c index cbfae5313d..d39fd7573c 100644 --- a/tools/rpmdump.c +++ b/tools/rpmdump.c @@ -48,6 +48,7 @@ static const char *sigTagName(uint32_t tag) case RPMSIGTAG_FILESIGNATURELENGTH: return "filesignaturelength"; case RPMSIGTAG_VERITYSIGNATURES: return "veritysignatures"; case RPMSIGTAG_VERITYSIGNATUREALGO: return "veritysignaturealgo"; + case RPMSIGTAG_RESERVED: return "Reserved"; default: break; } From 789838e1b8da50a35b6caed1c5d17a393757a083 Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 14:14:27 +0200 Subject: [PATCH 7/9] Add rpm package format version as a tag to >= v6 packages. It'd be harmless as v4 too, but adding it there serves no real purpose at this point and probably only complicates any logic to figure out a package version. With this, any package without that tag is not v6. --- build/pack.c | 3 +++ build/rpmbuild_internal.h | 2 +- include/rpm/rpmtag.h | 1 + tests/data/misc/rpmdump6.txt | 20 +++++++++++++------- tests/pinned/rpmsigdig6.txt | 2 +- tests/rpmgeneral.at | 1 + 6 files changed, 20 insertions(+), 9 deletions(-) diff --git a/build/pack.c b/build/pack.c index 7bf305b34b..b5bdb8c3b8 100644 --- a/build/pack.c +++ b/build/pack.c @@ -476,6 +476,9 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, headerPutString(pkg->header, RPMTAG_COOKIE, *cookie); } + if (pkg->rpmver >= 6) + headerPutUint32(pkg->header, RPMTAG_RPMFORMAT, &(pkg->rpmver), 1); + /* Create a dummy payload digests + size to get the header size right */ pld = nullDigest(pld_algo, 1); headerPutUint32(pkg->header, RPMTAG_PAYLOADDIGESTALGO, &pld_algo, 1); diff --git a/build/rpmbuild_internal.h b/build/rpmbuild_internal.h index 715b222790..a6a0d439d5 100644 --- a/build/rpmbuild_internal.h +++ b/build/rpmbuild_internal.h @@ -176,7 +176,7 @@ struct Package_s { rpmds dependencies[PACKAGE_NUM_DEPS]; rpmfiles cpioList; ARGV_t dpaths; - int rpmver; /* v4, v6? */ + unsigned int rpmver; /* v4, v6? */ struct Source * icon; diff --git a/include/rpm/rpmtag.h b/include/rpm/rpmtag.h index 2ac0b61c27..7560db8ad7 100644 --- a/include/rpm/rpmtag.h +++ b/include/rpm/rpmtag.h @@ -392,6 +392,7 @@ typedef enum rpmTag_e { RPMTAG_BUILDOPTION = 5111, /* internal */ RPMTAG_PAYLOADSIZE = 5112, /* l */ RPMTAG_PAYLOADSIZEALT = 5113, /* l */ + RPMTAG_RPMFORMAT = 5114, /* i */ RPMTAG_FIRSTFREE_TAG /*!< internal */ } rpmTag; diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index 61e2d92a26..cb74daef25 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -42,24 +42,24 @@ Tag #2 [region] Header: Header magic: 1e8ad8e (reserved: 0) -Index entries: 55 (880 bytes) -Data size: 2712 bytes -Header size: 3592 bytes +Index entries: 56 (896 bytes) +Data size: 2716 bytes +Header size: 3612 bytes -Region entries 55 -Region size 896 +Region entries 56 +Region size 912 Dribbles: 0 Tag #0 [region] tagno: 63 (Headerimmutable) type: 7 (blob) - offset: 2696 + offset: 2700 count: 16 region trailer tagno: 63 (Headerimmutable) type: 7 (blob) - offset: -880 + offset: -896 count: 16 Tag #1 [region] @@ -386,3 +386,9 @@ Tag #54 [region] offset: 2688 count: 1 +Tag #55 [region] + tagno: 5114 (Rpmformat) + type: 4 (int32) + offset: 2696 + count: 1 + diff --git a/tests/pinned/rpmsigdig6.txt b/tests/pinned/rpmsigdig6.txt index 898b9187a8..b63b424078 100644 --- a/tests/pinned/rpmsigdig6.txt +++ b/tests/pinned/rpmsigdig6.txt @@ -1,4 +1,4 @@ -SHA256HEADER: 37992f97f4116f6969eeffdc4dc41638630c5dedaa8ef0f466650aea326ec84a +SHA256HEADER: c62e9ba20111e91422c7cfb4c78b6e080ae5a8cfbc567dfd6e376e128d530baa SHA1HEADER: (none) SIGMD5: (none) PAYLOADDIGEST: 2f916d301d47ee34f16b74c7c49d11448fceb5487c6ba53c04950423fffeab95 diff --git a/tests/rpmgeneral.at b/tests/rpmgeneral.at index 8616a7d98f..0ab5ed6d44 100644 --- a/tests/rpmgeneral.at +++ b/tests/rpmgeneral.at @@ -263,6 +263,7 @@ REQUIRENAME REQUIRENEVRS REQUIRES REQUIREVERSION +RPMFORMAT RPMVERSION RSAHEADER SHA1HEADER From c6af742203fc0c4100501726866314a856af4bc5 Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 14:32:12 +0200 Subject: [PATCH 8/9] Don't populate os and arch in the lead structure The only thing that looks at these fields is "file", and even that would be better off not doing so because the canonical numbers are so meaningless: i386/x86_64 is the same thing, right? --- lib/rpmlead.c | 7 ------- tests/data/misc/rpmdump4.txt | 4 ++-- tests/data/misc/rpmdump6.txt | 4 ++-- 3 files changed, 4 insertions(+), 11 deletions(-) diff --git a/lib/rpmlead.c b/lib/rpmlead.c index d0db60a81d..d0d89d3a37 100644 --- a/lib/rpmlead.c +++ b/lib/rpmlead.c @@ -45,18 +45,11 @@ struct rpmlead_s { static int rpmLeadFromHeader(Header h, struct rpmlead_s *l) { if (h != NULL) { - int archnum, osnum; char * nevr = headerGetAsString(h, RPMTAG_NEVR); - /* FIXME: should grab these from header instead (RhBug:717898) */ - rpmGetArchInfo(NULL, &archnum); - rpmGetOsInfo(NULL, &osnum); - memset(l, 0, sizeof(*l)); l->major = 3; l->minor = 0; - l->archnum = archnum; - l->osnum = osnum; l->signature_type = RPMSIGTYPE_HEADERSIG; l->type = (headerIsSource(h) ? 1 : 0); diff --git a/tests/data/misc/rpmdump4.txt b/tests/data/misc/rpmdump4.txt index eeac859b21..d8ce1e8c45 100644 --- a/tests/data/misc/rpmdump4.txt +++ b/tests/data/misc/rpmdump4.txt @@ -2,8 +2,8 @@ Lead magic: edabeedb Version 3.0 Type: 0 Name: attrtest-1.0-1 -Arch: 1 -OS: 1 +Arch: 0 +OS: 0 Sigtype: 5 Signature: diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index cb74daef25..10abc8574b 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -2,8 +2,8 @@ Lead magic: edabeedb Version 3.0 Type: 0 Name: attrtest-1.0-1 -Arch: 1 -OS: 1 +Arch: 0 +OS: 0 Sigtype: 5 Signature: From bab2104dab8ead9db817653f1a70584faa19678b Mon Sep 17 00:00:00 2001 From: Panu Matilainen Date: Tue, 20 Feb 2024 14:37:43 +0200 Subject: [PATCH 9/9] Bump the rpm version in the lead to 4 for v6 packages Because rpm 4.x widely checks for the version in the lead to be either 3 or 4, bumping it to 6 would cause a total compatibility break for a value that is otherwise entirely unused in rpm. Which makes no sense whatsoever. As v3 and v4 used 3 as the major format (presumably for LSB compatibility reasons), we can at least bump it up to 4 for v6 to differentiate. It's not such a big lie as v6 uses the same fundamental structures as v4. For v7 we should be able to bump this up to the matching number... --- lib/rpmlead.c | 3 ++- tests/data/misc/rpmdump6.txt | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/rpmlead.c b/lib/rpmlead.c index d0d89d3a37..7420e3822e 100644 --- a/lib/rpmlead.c +++ b/lib/rpmlead.c @@ -48,7 +48,8 @@ static int rpmLeadFromHeader(Header h, struct rpmlead_s *l) char * nevr = headerGetAsString(h, RPMTAG_NEVR); memset(l, 0, sizeof(*l)); - l->major = 3; + /* v3 and v4 had 3 in the lead, use 4 for v6. Logical, eh? */ + l->major = headerIsEntry(h, RPMTAG_RPMFORMAT) ? 4 : 3; l->minor = 0; l->signature_type = RPMSIGTYPE_HEADERSIG; l->type = (headerIsSource(h) ? 1 : 0); diff --git a/tests/data/misc/rpmdump6.txt b/tests/data/misc/rpmdump6.txt index 10abc8574b..c73146b646 100644 --- a/tests/data/misc/rpmdump6.txt +++ b/tests/data/misc/rpmdump6.txt @@ -1,5 +1,5 @@ Lead magic: edabeedb -Version 3.0 +Version 4.0 Type: 0 Name: attrtest-1.0-1 Arch: 0