AVRO-1938: Add fingerprinting support to Python implementation #1181

subhashb · 2021-04-06T01:21:31Z

This PR adds support for generating fingerprints from schema objects. Schema fingerprints are extracted by invoking the fingerprint method on the schema object. By default, fingerprints are generated with the CRC-64 algorithm. Optionally, an algorithm, specified by the algorithm name used by hashlib, can be supplied.

All algorithms supported by hashlib are made available, but Avro recommends using one among CRC-32, MD5, and SHA256 as per needs.

Make sure you have checked all steps below.

Jira

My PR addresses the following Avro Jira issues and references them in the PR title.
- https://issues.apache.org/jira/browse/AVRO-1938

Tests

My PR adds the following unit tests:
- Fingerprint test cases in FingerprintTestCase class
- Miscellaneous fingerprint test cases in TestMisc class: test_unsupported_fingerprint_algorithm and test_less_popular_fingerprint_algorithm test methods

Commits

My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
Functionality confirms to Avro Specification.

With this change, Schema fingerprints can be extracted by invoking the `fingerprint` method on the schema object. By default, fingerprints will be generated with the CRC-64 algorithm. Optinally, the algorithm can be supplied. All algorithms supported by hashlib are available, but Avro recommends using one among CRC-32, MD5, and SHA256 as per needs.

lang/py/avro/schema.py

kojiromike · 2021-05-28T21:36:43Z

@subhashb In a recent PR I've moved the linting system over to use black. I guess if you autoformat your code with black it'll make the conflicts pretty easy to fix.

This commit addresses review comments and freezes the supported fingerprinting algorithms set.

subhashb · 2021-07-08T14:12:50Z

@kojiromike

Good on moving to black! 🎉 🎉
I have changed the code to use frozenset instead of set and fixed conflicts after running black.

kojiromike · 2021-07-13T02:10:23Z

@subhashb could you please take a crack at adding type hints to the changes here?

lang/py/avro/test/test_schema.py

lang/py/avro/schema.py

lang/py/avro/test/test_schema.py

Addresses PR 1181 review comments. Methods within Fingerprint mixin have been made available at the module level, including static variables used in fingerprinting. This PR has been synced with latest master.

subhashb · 2022-09-14T19:55:21Z

@kojiromike All comments have been addressed, along with type hints where applicable.

subhashb · 2022-12-12T18:56:13Z

@RyanSkraba @kojiromike Can you please help me understand what could be causing this error? Do we need to flush caches?

As fas as I can see, these failures are not due to changes in this PR's branch.

martin-g · 2022-12-12T20:28:29Z

There is some problem with Python since recently (today ?!).
It fails even on master branch with these errors: https://github.com/apache/avro/actions/workflows/test-lang-py.yml

subhashb · 2022-12-12T21:42:25Z

@martin-g Could it have something to do with this change: pytest-dev/pytest-xdist#821 ?

tox uses pytest-xdist internally. I am unsure; the last pytest-xdist release was 11 days ago. Either way, we may have to clear the cache at the repo level and retry. I reinitialized my virtualenv, and tests still succeed, so I assume the error is because of broken dependencies.

@RyanSkraba If you agree, we will need to clear the cache manually.

subhashb · 2023-07-16T17:12:45Z

@kojiromike Synced with latest master. Can you please approve workflow runs and review the PR?

lang/py/avro/schema.py

lang/py/avro/test/test_schema.py

subhashb · 2023-07-17T00:04:17Z

@kojiromike Addressed review comments.

lang/py/avro/schema.py

lang/py/avro/test/test_schema.py

subhashb · 2023-07-17T05:36:11Z

@kojiromike Addressed further review comments and left a note about try/except format instead of assertRaises.

@kojiromike

* AVRO-1938 Add support for fingerprinting schemas With this change, Schema fingerprints can be extracted by invoking the `fingerprint` method on the schema object. By default, fingerprints will be generated with the CRC-64 algorithm. Optinally, the algorithm can be supplied. All algorithms supported by hashlib are available, but Avro recommends using one among CRC-32, MD5, and SHA256 as per needs. * AVRO-1938 Fix issue with AbstractSet typecheck * Format with black * Freeze Supported Algorithms Set This commit addresses review comments and freezes the supported fingerprinting algorithms set. * Minor lint fix with black * Address Typecheck issues with Frozenset * Fold Fingerprint Mixin within Schema Addresses PR 1181 review comments. Methods within Fingerprint mixin have been made available at the module level, including static variables used in fingerprinting. This PR has been synced with latest master. * Add type hints to fingerprint methods/variables * Fix incorrect import sorting in schema.py to pass lint check * Address @kojiromike Jul 16 review comments * Address @kojiromike Jul 16 review comments - 2 * Address @kojiromike Jul 17 review comments * Fix black lint issue (cherry picked from commit f504265)

@kojiromike

…e#1181) * AVRO-1938 Add support for fingerprinting schemas With this change, Schema fingerprints can be extracted by invoking the `fingerprint` method on the schema object. By default, fingerprints will be generated with the CRC-64 algorithm. Optinally, the algorithm can be supplied. All algorithms supported by hashlib are available, but Avro recommends using one among CRC-32, MD5, and SHA256 as per needs. * AVRO-1938 Fix issue with AbstractSet typecheck * Format with black * Freeze Supported Algorithms Set This commit addresses review comments and freezes the supported fingerprinting algorithms set. * Minor lint fix with black * Address Typecheck issues with Frozenset * Fold Fingerprint Mixin within Schema Addresses PR 1181 review comments. Methods within Fingerprint mixin have been made available at the module level, including static variables used in fingerprinting. This PR has been synced with latest master. * Add type hints to fingerprint methods/variables * Fix incorrect import sorting in schema.py to pass lint check * Address @kojiromike Jul 16 review comments * Address @kojiromike Jul 16 review comments - 2 * Address @kojiromike Jul 17 review comments * Fix black lint issue

github-actions bot added the Python label Apr 6, 2021

AVRO-1938 Fix issue with AbstractSet typecheck

2963871

subhashb commented Apr 6, 2021

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

subhashb changed the title ~~AVRO-1938: Add support for fingerprinting schemas~~ AVRO-1938: Add fingerprinting support to Python implementation Apr 6, 2021

subhashb added 5 commits July 7, 2021 16:59

Merge branch 'master' into 1938-add-fingerprint-support

f2c2a8e

Format with black

58e8afc

Freeze Supported Algorithms Set

6ad0038

This commit addresses review comments and freezes the supported fingerprinting algorithms set.

Minor lint fix with black

d522bbc

Address Typecheck issues with Frozenset

99c4998

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 15, 2021

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

Merge branch 'master' into 1938-add-fingerprint-support

e5ee1f2

github-advanced-security bot found potential problems Sep 13, 2022

View reviewed changes

lang/py/avro/test/test_schema.py Fixed Show fixed Hide fixed

lang/py/avro/test/test_schema.py Fixed Show fixed Hide fixed

subhashb added 2 commits September 14, 2022 11:17

Fold Fingerprint Mixin within Schema

f409188

Addresses PR 1181 review comments. Methods within Fingerprint mixin have been made available at the module level, including static variables used in fingerprinting. This PR has been synced with latest master.

Add type hints to fingerprint methods/variables

0307169

subhashb added 2 commits December 12, 2022 09:52

Merge branch 'master' into 1938-add-fingerprint-support

578a6e8

Fix incorrect import sorting in schema.py to pass lint check

a53f8e1

Merge branch 'master' into 1938-add-fingerprint-support

ebd48be

kojiromike reviewed Jul 16, 2023

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 16, 2023

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 16, 2023

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

Address @kojiromike Jul 16 review comments

65711b3

kojiromike reviewed Jul 17, 2023

View reviewed changes

lang/py/avro/schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 17, 2023

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

kojiromike reviewed Jul 17, 2023

View reviewed changes

lang/py/avro/test/test_schema.py Outdated Show resolved Hide resolved

Address @kojiromike Jul 16 review comments - 2

47ed33a

Address @kojiromike Jul 17 review comments

421b4da

kojiromike approved these changes Jul 17, 2023

View reviewed changes

Fix black lint issue

d5191ba

kojiromike merged commit f504265 into apache:master Jul 17, 2023
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AVRO-1938: Add fingerprinting support to Python implementation #1181

AVRO-1938: Add fingerprinting support to Python implementation #1181

subhashb commented Apr 6, 2021

kojiromike commented May 28, 2021

subhashb commented Jul 8, 2021

kojiromike commented Jul 13, 2021

subhashb commented Sep 14, 2022

subhashb commented Dec 12, 2022

martin-g commented Dec 12, 2022

subhashb commented Dec 12, 2022

subhashb commented Jul 16, 2023

subhashb commented Jul 17, 2023

subhashb commented Jul 17, 2023

AVRO-1938: Add fingerprinting support to Python implementation #1181

AVRO-1938: Add fingerprinting support to Python implementation #1181

Conversation

subhashb commented Apr 6, 2021

Jira

Tests

Commits

Documentation

kojiromike commented May 28, 2021

subhashb commented Jul 8, 2021

kojiromike commented Jul 13, 2021

subhashb commented Sep 14, 2022

subhashb commented Dec 12, 2022

martin-g commented Dec 12, 2022

subhashb commented Dec 12, 2022

subhashb commented Jul 16, 2023

subhashb commented Jul 17, 2023

subhashb commented Jul 17, 2023