Skip to content

KAFKA-20559: Update NOTICE-binary and add verify_notice.py#22266

Open
majialoong wants to merge 3 commits into
apache:trunkfrom
majialoong:KAFKA-20559
Open

KAFKA-20559: Update NOTICE-binary and add verify_notice.py#22266
majialoong wants to merge 3 commits into
apache:trunkfrom
majialoong:KAFKA-20559

Conversation

@majialoong
Copy link
Copy Markdown
Contributor

@majialoong majialoong commented May 12, 2026

Update NOTICE-binary and add verify_notice.py

verify_notice.py checks that NOTICE-binarycontains every NOTICE text embedded in the third-party jars shipped by releaseTarGz, ensuring all upstream attribution notices are included in the distribution as required by the Apache License 2.0.

It extracts all NOTICE-like files from third-party jars, deduplicates them by normalised content, then searches the third-party body of NOTICE-binary for each one using a whitespace-tolerant match. It reports two lists and exits non-zero if either is non-empty:

  • Unmatched – upstream NOTICEs missing from NOTICE-binary (must be added).
  • Leftover – content in NOTICE-binary not claimed by any upstream NOTICE (potentially stale).

@github-actions github-actions Bot added tools triage PRs from the community labels May 12, 2026
@majialoong
Copy link
Copy Markdown
Contributor Author

majialoong commented May 12, 2026

The verify_notice.py output will look something like this:

Kafka-own jars excluded by regex '(kafka|connect|trogdor)' (29):
  - connect-api-4.4.0-SNAPSHOT.jar
  - connect-basic-auth-extension-4.4.0-SNAPSHOT.jar
  - connect-file-4.4.0-SNAPSHOT.jar
  - connect-json-4.4.0-SNAPSHOT.jar
  - connect-mirror-4.4.0-SNAPSHOT.jar
  - connect-mirror-client-4.4.0-SNAPSHOT.jar
  - connect-runtime-4.4.0-SNAPSHOT.jar
  - connect-transforms-4.4.0-SNAPSHOT.jar
  - kafka-clients-4.4.0-SNAPSHOT.jar
  - kafka-coordinator-common-4.4.0-SNAPSHOT.jar
  - kafka-group-coordinator-4.4.0-SNAPSHOT.jar
  - kafka-group-coordinator-api-4.4.0-SNAPSHOT.jar
  - kafka-metadata-4.4.0-SNAPSHOT.jar
  - kafka-raft-4.4.0-SNAPSHOT.jar
  - kafka-server-4.4.0-SNAPSHOT.jar
  - kafka-server-common-4.4.0-SNAPSHOT.jar
  - kafka-share-coordinator-4.4.0-SNAPSHOT.jar
  - kafka-shell-4.4.0-SNAPSHOT.jar
  - kafka-storage-4.4.0-SNAPSHOT.jar
  - kafka-storage-api-4.4.0-SNAPSHOT.jar
  - kafka-streams-4.4.0-SNAPSHOT.jar
  - kafka-streams-examples-4.4.0-SNAPSHOT.jar
  - kafka-streams-scala_2.13-4.4.0-SNAPSHOT.jar
  - kafka-streams-test-utils-4.4.0-SNAPSHOT.jar
  - kafka-tools-4.4.0-SNAPSHOT.jar
  - kafka-tools-api-4.4.0-SNAPSHOT.jar
  - kafka-transaction-coordinator-4.4.0-SNAPSHOT.jar
  - kafka_2.13-4.4.0-SNAPSHOT.jar
  - trogdor-4.4.0-SNAPSHOT.jar

Third-party jars to inspect (78):
  - HdrHistogram-2.2.2.jar
  - activation-1.1.1.jar
  - aopalliance-repackaged-3.0.6.jar
  - argparse4j-0.7.0.jar
  - caffeine-3.2.0.jar
  - classgraph-4.8.179.jar
  - commons-beanutils-1.11.0.jar
  - commons-collections-3.2.2.jar
  - commons-digester-2.1.jar
  - commons-logging-1.3.5.jar
  - commons-validator-1.10.1.jar
  - hash4j-0.22.0.jar
  - hk2-api-3.0.6.jar
  - hk2-locator-3.0.6.jar
  - hk2-utils-3.0.6.jar
  - jackson-annotations-2.21.jar
  - jackson-core-2.21.2.jar
  - jackson-databind-2.21.2.jar
  - jackson-dataformat-csv-2.21.2.jar
  - jackson-dataformat-yaml-2.21.2.jar
  - jackson-datatype-jdk8-2.21.2.jar
  - jackson-jakarta-rs-base-2.21.2.jar
  - jackson-jakarta-rs-json-provider-2.21.2.jar
  - jackson-module-blackbird-2.21.2.jar
  - jackson-module-jakarta-xmlbind-annotations-2.21.2.jar
  - jakarta.activation-2.0.1.jar
  - jakarta.activation-api-2.1.0.jar
  - jakarta.annotation-api-2.1.1.jar
  - jakarta.inject-api-2.0.1.jar
  - jakarta.servlet-api-6.0.0.jar
  - jakarta.validation-api-3.0.2.jar
  - jakarta.ws.rs-api-3.1.0.jar
  - jakarta.xml.bind-api-3.0.1.jar
  - javassist-3.30.2-GA.jar
  - javax.activation-api-1.2.0.jar
  - jaxb-api-2.3.1.jar
  - jersey-client-3.1.10.jar
  - jersey-common-3.1.10.jar
  - jersey-container-servlet-3.1.10.jar
  - jersey-container-servlet-core-3.1.10.jar
  - jersey-hk2-3.1.10.jar
  - jersey-server-3.1.10.jar
  - jetty-alpn-client-12.0.34.jar
  - jetty-client-12.0.34.jar
  - jetty-ee10-servlet-12.0.34.jar
  - jetty-ee10-servlets-12.0.34.jar
  - jetty-http-12.0.34.jar
  - jetty-io-12.0.34.jar
  - jetty-security-12.0.34.jar
  - jetty-server-12.0.34.jar
  - jetty-session-12.0.34.jar
  - jetty-util-12.0.34.jar
  - jline-3.30.4.jar
  - jopt-simple-5.0.4.jar
  - jose4j-0.9.6.jar
  - jspecify-1.0.0.jar
  - log4j-1.2-api-2.25.4.jar
  - log4j-api-2.25.4.jar
  - log4j-core-2.25.4.jar
  - log4j-slf4j-impl-2.25.4.jar
  - lz4-java-1.10.2.jar
  - maven-artifact-3.9.15.jar
  - metrics-core-2.2.0.jar
  - opentelemetry-proto-1.3.2-alpha.jar
  - osgi-resource-locator-1.0.3.jar
  - pcollections-4.0.2.jar
  - plexus-utils-3.6.1.jar
  - protobuf-java-3.25.5.jar
  - re2j-1.8.jar
  - rocksdbjni-10.1.3.jar
  - scala-library-2.13.18.jar
  - scala-logging_2.13-3.9.6.jar
  - scala-reflect-2.13.18.jar
  - slf4j-api-1.7.36.jar
  - snakeyaml-2.5.jar
  - snappy-java-1.1.10.7.jar
  - swagger-annotations-2.2.48.jar
  - zstd-jni-1.5.6-10.jar

Jars containing a NOTICE file (38):
  + aopalliance-repackaged-3.0.6.jar  [META-INF/NOTICE.md]
  + commons-beanutils-1.11.0.jar  [META-INF/NOTICE.txt]
  + commons-collections-3.2.2.jar  [META-INF/NOTICE.txt]
  + commons-digester-2.1.jar  [META-INF/NOTICE.txt]
  + commons-logging-1.3.5.jar  [META-INF/NOTICE.txt]
  + commons-validator-1.10.1.jar  [META-INF/NOTICE.txt]
  + hk2-api-3.0.6.jar  [META-INF/NOTICE.md]
  + hk2-locator-3.0.6.jar  [META-INF/NOTICE.md]
  + hk2-utils-3.0.6.jar  [META-INF/NOTICE.md]
  + jackson-annotations-2.21.jar  [META-INF/NOTICE]
  + jackson-core-2.21.2.jar  [META-INF/NOTICE, META-INF/FastDoubleParser-NOTICE]
  + jackson-databind-2.21.2.jar  [META-INF/NOTICE]
  + jackson-dataformat-csv-2.21.2.jar  [META-INF/NOTICE]
  + jackson-dataformat-yaml-2.21.2.jar  [META-INF/NOTICE]
  + jackson-datatype-jdk8-2.21.2.jar  [META-INF/NOTICE]
  + jackson-jakarta-rs-json-provider-2.21.2.jar  [META-INF/NOTICE]
  + jackson-module-blackbird-2.21.2.jar  [META-INF/NOTICE]
  + jakarta.activation-2.0.1.jar  [META-INF/NOTICE.md]
  + jakarta.activation-api-2.1.0.jar  [META-INF/NOTICE.md]
  + jakarta.annotation-api-2.1.1.jar  [META-INF/NOTICE.md]
  + jakarta.inject-api-2.0.1.jar  [META-INF/NOTICE.md]
  + jakarta.ws.rs-api-3.1.0.jar  [META-INF/NOTICE.md]
  + jakarta.xml.bind-api-3.0.1.jar  [META-INF/NOTICE.md]
  + jersey-client-3.1.10.jar  [META-INF/NOTICE.md]
  + jersey-common-3.1.10.jar  [META-INF/NOTICE.markdown, META-INF/NOTICE.md]
  + jersey-container-servlet-3.1.10.jar  [META-INF/NOTICE.md]
  + jersey-container-servlet-core-3.1.10.jar  [META-INF/NOTICE.md]
  + jersey-hk2-3.1.10.jar  [META-INF/NOTICE.md]
  + jersey-server-3.1.10.jar  [META-INF/NOTICE.markdown, META-INF/NOTICE.md]
  + log4j-1.2-api-2.25.4.jar  [META-INF/NOTICE]
  + log4j-api-2.25.4.jar  [META-INF/NOTICE]
  + log4j-core-2.25.4.jar  [META-INF/NOTICE]
  + log4j-slf4j-impl-2.25.4.jar  [META-INF/NOTICE]
  + maven-artifact-3.9.15.jar  [META-INF/NOTICE]
  + plexus-utils-3.6.1.jar  [META-INF/NOTICE]
  + scala-library-2.13.18.jar  [NOTICE]
  + scala-reflect-2.13.18.jar  [NOTICE]
  + swagger-annotations-2.2.48.jar  [META-INF/NOTICE]

Jars WITHOUT a NOTICE file (40):
  - HdrHistogram-2.2.2.jar
  - activation-1.1.1.jar
  - argparse4j-0.7.0.jar
  - caffeine-3.2.0.jar
  - classgraph-4.8.179.jar
  - hash4j-0.22.0.jar
  - jackson-jakarta-rs-base-2.21.2.jar
  - jackson-module-jakarta-xmlbind-annotations-2.21.2.jar
  - jakarta.servlet-api-6.0.0.jar
  - jakarta.validation-api-3.0.2.jar
  - javassist-3.30.2-GA.jar
  - javax.activation-api-1.2.0.jar
  - jaxb-api-2.3.1.jar
  - jetty-alpn-client-12.0.34.jar
  - jetty-client-12.0.34.jar
  - jetty-ee10-servlet-12.0.34.jar
  - jetty-ee10-servlets-12.0.34.jar
  - jetty-http-12.0.34.jar
  - jetty-io-12.0.34.jar
  - jetty-security-12.0.34.jar
  - jetty-server-12.0.34.jar
  - jetty-session-12.0.34.jar
  - jetty-util-12.0.34.jar
  - jline-3.30.4.jar
  - jopt-simple-5.0.4.jar
  - jose4j-0.9.6.jar
  - jspecify-1.0.0.jar
  - lz4-java-1.10.2.jar
  - metrics-core-2.2.0.jar
  - opentelemetry-proto-1.3.2-alpha.jar
  - osgi-resource-locator-1.0.3.jar
  - pcollections-4.0.2.jar
  - protobuf-java-3.25.5.jar
  - re2j-1.8.jar
  - rocksdbjni-10.1.3.jar
  - scala-logging_2.13-3.9.6.jar
  - slf4j-api-1.7.36.jar
  - snakeyaml-2.5.jar
  - snappy-java-1.1.10.7.jar
  - zstd-jni-1.5.6-10.jar

NOTICE de-duplication: 41 NOTICE file(s) across 38 jar(s) -> 29 unique NOTICE text(s) (6 shared group(s) + 23 singleton(s)).

Shared NOTICE groups (6):
  * aopalliance-repackaged-3.0.6.jar, hk2-api-3.0.6.jar, hk2-locator-3.0.6.jar, hk2-utils-3.0.6.jar
      - aopalliance-repackaged-3.0.6.jar  (META-INF/NOTICE.md)
      - hk2-api-3.0.6.jar  (META-INF/NOTICE.md)
      - hk2-locator-3.0.6.jar  (META-INF/NOTICE.md)
      - hk2-utils-3.0.6.jar  (META-INF/NOTICE.md)
  * jackson-annotations-2.21.jar, jackson-databind-2.21.2.jar
      - jackson-annotations-2.21.jar  (META-INF/NOTICE)
      - jackson-databind-2.21.2.jar  (META-INF/NOTICE)
  * jackson-dataformat-csv-2.21.2.jar, jackson-dataformat-yaml-2.21.2.jar
      - jackson-dataformat-csv-2.21.2.jar  (META-INF/NOTICE)
      - jackson-dataformat-yaml-2.21.2.jar  (META-INF/NOTICE)
  * jakarta.activation-2.0.1.jar, jakarta.activation-api-2.1.0.jar
      - jakarta.activation-2.0.1.jar  (META-INF/NOTICE.md)
      - jakarta.activation-api-2.1.0.jar  (META-INF/NOTICE.md)
  * jersey-client-3.1.10.jar, jersey-common-3.1.10.jar, jersey-container-servlet-3.1.10.jar, jersey-container-servlet-core-3.1.10.jar, jersey-hk2-3.1.10.jar, jersey-server-3.1.10.jar
      - jersey-client-3.1.10.jar  (META-INF/NOTICE.md)
      - jersey-common-3.1.10.jar  (META-INF/NOTICE.md)
      - jersey-container-servlet-3.1.10.jar  (META-INF/NOTICE.md)
      - jersey-container-servlet-core-3.1.10.jar  (META-INF/NOTICE.md)
      - jersey-hk2-3.1.10.jar  (META-INF/NOTICE.md)
      - jersey-server-3.1.10.jar  (META-INF/NOTICE.md)
  * scala-library-2.13.18.jar, scala-reflect-2.13.18.jar
      - scala-library-2.13.18.jar  (NOTICE)
      - scala-reflect-2.13.18.jar  (NOTICE)

NOTICE-binary header preserved (1129 chars); third-party body to validate (27574 chars, starting at line 24).

Matched upstream NOTICEs already present in NOTICE-binary (28):
  [01] OK  source(s):
          - jersey-client-3.1.10.jar  (META-INF/NOTICE.md)
          - jersey-common-3.1.10.jar  (META-INF/NOTICE.md)
          - jersey-container-servlet-3.1.10.jar  (META-INF/NOTICE.md)
          - jersey-container-servlet-core-3.1.10.jar  (META-INF/NOTICE.md)
          - jersey-hk2-3.1.10.jar  (META-INF/NOTICE.md)
          - jersey-server-3.1.10.jar  (META-INF/NOTICE.md)
  [02] OK  source(s):
          - jakarta.xml.bind-api-3.0.1.jar  (META-INF/NOTICE.md)
  [03] OK  source(s):
          - aopalliance-repackaged-3.0.6.jar  (META-INF/NOTICE.md)
          - hk2-api-3.0.6.jar  (META-INF/NOTICE.md)
          - hk2-locator-3.0.6.jar  (META-INF/NOTICE.md)
          - hk2-utils-3.0.6.jar  (META-INF/NOTICE.md)
  [04] OK  source(s):
          - jackson-core-2.21.2.jar  (META-INF/FastDoubleParser-NOTICE)
  [05] OK  source(s):
          - jakarta.ws.rs-api-3.1.0.jar  (META-INF/NOTICE.md)
  [06] OK  source(s):
          - jersey-server-3.1.10.jar  (META-INF/NOTICE.markdown)
  [07] OK  source(s):
          - jersey-common-3.1.10.jar  (META-INF/NOTICE.markdown)
  [08] OK  source(s):
          - jakarta.annotation-api-2.1.1.jar  (META-INF/NOTICE.md)
  [09] OK  source(s):
          - jackson-core-2.21.2.jar  (META-INF/NOTICE)
  [10] OK  source(s):
          - jakarta.inject-api-2.0.1.jar  (META-INF/NOTICE.md)
  [11] OK  source(s):
          - jackson-module-blackbird-2.21.2.jar  (META-INF/NOTICE)
  [12] OK  source(s):
          - jackson-jakarta-rs-json-provider-2.21.2.jar  (META-INF/NOTICE)
  [13] OK  source(s):
          - jakarta.activation-2.0.1.jar  (META-INF/NOTICE.md)
          - jakarta.activation-api-2.1.0.jar  (META-INF/NOTICE.md)
  [14] OK  source(s):
          - jackson-annotations-2.21.jar  (META-INF/NOTICE)
          - jackson-databind-2.21.2.jar  (META-INF/NOTICE)
  [15] OK  source(s):
          - jackson-dataformat-csv-2.21.2.jar  (META-INF/NOTICE)
          - jackson-dataformat-yaml-2.21.2.jar  (META-INF/NOTICE)
  [16] OK  source(s):
          - scala-library-2.13.18.jar  (NOTICE)
          - scala-reflect-2.13.18.jar  (NOTICE)
  [17] OK  source(s):
          - jackson-datatype-jdk8-2.21.2.jar  (META-INF/NOTICE)
  [18] OK  source(s):
          - plexus-utils-3.6.1.jar  (META-INF/NOTICE)
  [19] OK  source(s):
          - log4j-core-2.25.4.jar  (META-INF/NOTICE)
  [20] OK  source(s):
          - swagger-annotations-2.2.48.jar  (META-INF/NOTICE)
  [21] OK  source(s):
          - log4j-1.2-api-2.25.4.jar  (META-INF/NOTICE)
  [22] OK  source(s):
          - log4j-slf4j-impl-2.25.4.jar  (META-INF/NOTICE)
  [23] OK  source(s):
          - commons-collections-3.2.2.jar  (META-INF/NOTICE.txt)
  [24] OK  source(s):
          - commons-validator-1.10.1.jar  (META-INF/NOTICE.txt)
  [25] OK  source(s):
          - commons-digester-2.1.jar  (META-INF/NOTICE.txt)
  [26] OK  source(s):
          - commons-logging-1.3.5.jar  (META-INF/NOTICE.txt)
  [27] OK  source(s):
          - log4j-api-2.25.4.jar  (META-INF/NOTICE)
  [28] OK  source(s):
          - maven-artifact-3.9.15.jar  (META-INF/NOTICE)

Upstream NOTICEs NOT matched in NOTICE-binary (1 of 29 unique upstream NOTICEs):

------------------------------------------------------------------------------
  [01] source(s):
          - commons-beanutils-1.11.0.jar  (META-INF/NOTICE.txt)
------------------------------------------------------------------------------
  | Apache Commons BeanUtils
  | Copyright 2000-2025 The Apache Software Foundation

  | This product includes software developed at
  | The Apache Software Foundation (https://www.apache.org/).

Leftover content in NOTICE-binary, potentially stale (1 block(s)):

------------------------------------------------------------------------------
  [01] NOTICE-binary lines L83-L87
------------------------------------------------------------------------------
    83 | Apache Commons BeanUtils
    84 | Copyright 2000-2024 The Apache Software Foundation
    85 | 
    86 | This product includes software developed at
    87 | The Apache Software Foundation (https://www.apache.org/).

NOTICE-binary needs human review: upstream NOTICEs listed above must be added, and leftover blocks should be re-checked.

@chia7712
Copy link
Copy Markdown
Member

chia7712 commented May 18, 2026

@majialoong would you mind intentionally making a small mismatch to test the CI?

@github-actions github-actions Bot added the build Gradle build or GitHub Actions label May 18, 2026
@majialoong
Copy link
Copy Markdown
Contributor Author

majialoong commented May 18, 2026

@chia7712 Done! Integrated verify_notice.py into the CI and added an intentional error in NOTICE-binary to test the pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

build Gradle build or GitHub Actions ci-approved tools triage PRs from the community

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@majialoong @chia7712 @m1a2st