pandaproxy/sr: Re-purpose schema_registry_normalize_on_startup config

[IoannisRP]

Implements: CORE-9526

The new functionality allows to blanket-enable schema normalization in every SR operation.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v25.1.x
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

• none

[vbotbuildovich]

CI test results

test results on build#63437

test_id	test_kind	job_url	test_status	passed
rptest.tests.datalake.batching_test.DatalakeBatchingTest.test_batching.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.catalog_type=CatalogType.REST_JDBC.expect_large_files=True	ducktape	https://buildkite.com/redpanda/redpanda/builds/63437#0195b435-5dcd-44b3-8512-3691cfb40706	FLAKY	1/2
rptest.tests.datalake.compaction_test.CompactionTest.test_compaction.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC	ducktape	https://buildkite.com/redpanda/redpanda/builds/63437#0195b435-5dd0-4672-b43b-fe9bcd49db2a	FLAKY	1/2
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE	ducktape	https://buildkite.com/redpanda/redpanda/builds/63437#0195b432-b329-48d1-aac0-a01ba06889a7	FLAKY	1/2

test results on build#63493

test_id	test_kind	job_url	test_status	passed
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=.cancellation.dir.in.stage.preparing.use_alias.True	ducktape	https://buildkite.com/redpanda/redpanda/builds/63493#0195b875-67c5-44ed-b517-82b4c7ba75fc	FLAKY	1/2
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=.cancellation.dir.in.stage.preparing.use_alias.True	ducktape	https://buildkite.com/redpanda/redpanda/builds/63493#0195b875-67c7-422a-b366-9d30e6b37e50	FLAKY	1/3
rptest.tests.datalake.datalake_upgrade_test.DatalakeUpgradeTest.test_upload_through_upgrade.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK	ducktape	https://buildkite.com/redpanda/redpanda/builds/63493#0195b878-5cb5-4a11-adb6-b82f2f4120e3	FLAKY	1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic	ducktape	https://buildkite.com/redpanda/redpanda/builds/63493#0195b878-5cb5-4a11-adb6-b82f2f4120e3	FLAKY	1/2

[kbatuigas]

Just a suggestion for clarity in docs 👍

[IoannisRP]

Changes in force-push:

• Updated schema_registry_always_normalize description

[BenPope]

I see no reason not to make this configurable at runtime.

[IoannisRP]

There is no intrinsic reason as to why we can't flip it, apart from ease of change.

We call the sharded_store constructor in quite a few places, and all of them would need either a binding or a mock_binding for tests. It would add lot's of noise to the pr. I would prefer to make up a follow up for that, with just that change.

[vbotbuildovich]

/backport v25.1.x

[vbotbuildovich]

/backport v24.3.x

[vbotbuildovich]

/backport v24.2.x

[vbotbuildovich]

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-25429-v24.3.x-308 remotes/upstream/v24.3.x
git cherry-pick -x a17d27353d

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v24.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-25429-v24.2.x-324 remotes/upstream/v24.2.x
git cherry-pick -x a17d27353d

Workflow run logs.

[dotnwat]

@IoannisRP @BenPope is the expectation that schema_registry_always_normalize will take on the previous value of schema_registry_normalize_on_startup after upgrade because you've set .aliases? If so, have you checked, because I don't think behavior exists.

[IoannisRP]

@dotnwat previous behavior was kinda trampled by #25217 and reintroduced here.

[dotnwat]

@IoannisRP i'm having trouble understanding how that PR relates to my question. can you explain?

[IoannisRP]

sorry, misunderstood the question. Yeah, just checked. It does respect the old value after an update.

[dotnwat]

sorry, misunderstood the question. Yeah, just checked. It does respect the old value after an update.

@IoannisRP so i still have the same question: what is the expectation? is the expectation that if schema_registry_normalize_on_startup were set to true (a non-default value) prior to upgrading to this code, that the new configuration would also be true? if so, i don't think that it works that way. did you test that scenario?

[IoannisRP]

is the expectation that if schema_registry_normalize_on_startup were set to true (a non-default value) prior to upgrading to this code, that the new configuration would also be true?

yes, the expectation is that the new config will respect the potentially non-default value of the old alias.

if so, i don't think that it works that way. did you test that scenario?

it did test this scenario, yes. I run an older rp without my changes and set schema_registry_normalize_on_startup to true. Then I run a rp with my changes and it reports schema_registry_always_normalize as true

You can also check any other PR where an alias replaces an existing config. They all just rename the variable and keep the old name as a part in .aliases.

[dotnwat]

@IoannisRP huh, indeed. i tried it again and it does seem to be working this way. im not sure how i messed up the test before. sorry for the noise!