Engine: Plan to clean up from deprecated interfaces and to transit to OpenSSL 4.0

[VladGud]

Engine: Plan to clean up from deprecated interfaces and to transit to OpenSSL 4.0

This issue tracks the work required to fully decouple the GOST provider codebase from deprecated OpenSSL interfaces and prepare it for future OpenSSL releases.

Scope

The goal is to eliminate usage of deprecated ENGINE- and EVP-based APIs, simplify provider initialization, and ensure compatibility with OpenSSL builds that disable deprecated functionality.

Deprecated OpenSSL Interfaces

The [Deprecated OpenSSL Interface](https://docs.openssl.org/master/man7/ossl-guide-migration/#deprecated-function-mappings) section provides a complete list of interfaces marked as Deprecated which, at the time of analysis, were still supported in OpenSSL 4.0, but could be removed in subsequent releases.

The gost-engine source code was analyzed to identify which of the deprecated and removed OpenSSL interfaces are used.
For each identified case, a migration strategy was defined.

Deprecated MAC Methods

№	Deprecated method	Where found (file:lines)	Proposed replacement
1	CMAC_CTX_copy	gost_omac.c:122	Migrate to EVP_MAC (CMAC) and duplicate the context using EVP_MAC_CTX_dup() (if available).
2	CMAC_CTX_free	gost_omac.c:114,131,142	Migrate to EVP_MAC_CTX_free() and EVP_MAC_free() (if an EVP_MAC* is stored).
3	CMAC_CTX_new	gost_omac.c:120,143	Migrate to EVP_MAC_fetch("CMAC") + EVP_MAC_CTX_new().
4	CMAC_Final	gost_omac.c:93	Migrate to EVP_MAC_final() (write output to a buffer, then memcpy to dgst_size, as currently done).
5	CMAC_Init	gost_omac.c:149	Migrate to EVP_MAC_init() with parameter OSSL_MAC_PARAM_CIPHER (cipher specified by name, e.g. c->cipher_name).
6	CMAC_Update	gost_omac.c:79	Migrate to EVP_MAC_update().
7	HMAC_CTX_free	gost_keyexpimp.c:250,258	Migrate to EVP_MAC (HMAC): free via EVP_MAC_CTX_free() and EVP_MAC_free().
8	HMAC_CTX_new	gost_keyexpimp.c:217	Migrate to EVP_MAC_fetch("HMAC") + EVP_MAC_CTX_new().
9	HMAC_CTX_reset	gost_keyexpimp.c:254	For EVP_MAC, the equivalent of reset is re-running EVP_MAC_init() with the same key and parameters.
10	HMAC_Final	gost_keyexpimp.c:248	Migrate to EVP_MAC_final().
11	HMAC_Init_ex	gost_keyexpimp.c:240	Migrate to EVP_MAC_init() with parameters OSSL_MAC_PARAM_DIGEST and the key.
12	HMAC_Update	gost_keyexpimp.c:243–247	Migrate to EVP_MAC_update().

---

Deprecated EC_KEY Interfaces

№	Deprecated method	Where found (file:lines)	Proposed replacement
13	EC_KEY_check_key	gost_ameth.c:203; gost_pmeth.c:1147	Delegate to OpenSSL or fallback COMPAT_EC_KEY implementation.
14	EC_KEY_free	Multiple files	Unified deallocation via OpenSSL or COMPAT_EC_KEY.
15	EC_KEY_get0_group	Multiple files	Access group via OpenSSL or compat implementation.
16	EC_KEY_get0_private_key	Multiple files	Retrieve private key via OpenSSL or compat implementation.
17	EC_KEY_get0_public_key	Multiple files	Retrieve public key via OpenSSL or compat implementation.
18	EC_KEY_new	Multiple files	Create key via OpenSSL or initialize COMPAT_EC_KEY.
19	EC_KEY_set_group	gost_ameth.c:1058; others	Assign group via OpenSSL or store in compat structure.
20	EC_KEY_set_private_key	gost_ameth.c:199; others	Assign private key via OpenSSL or compat structure.
21	EC_KEY_set_public_key	gost_ameth.c:447,1132; others	Assign public key via OpenSSL or compat structure.

---

Deprecated EVP_CIPHER_CTX Accessors

№	Deprecated method	Where found (file:lines)	Proposed replacement
22	EVP_CIPHER_CTX_iv	gost_crypt.c:1409; others	Store IV in custom context (e.g. gost_ctx).
23	EVP_CIPHER_CTX_iv_noconst	Multiple files	Do not modify internal IV; use custom state.
24	EVP_CIPHER_CTX_original_iv	Multiple files	Store original IV in custom context.

---

Deprecated EVP_PKEY Accessors

№	Deprecated method	Where found (file:lines)	Proposed replacement
25	EVP_PKEY_assign	Multiple files	Move to _legacy files, compile with -DGOST_ENABLE_LEGACY=1.
26	EVP_PKEY_get0	Multiple files	Same as above.

---

Removed Interfaces and Migration Strategy

(OpenSSL 4.0)

EVP_CIPHER_meth_* (removed)

№	Removed interface	Where found (file:lines)	Migration strategy
27	EVP_CIPHER_meth_new	gost_crypt.c:117	Move to gost_crypt_legacy.c, conditional build.
28	EVP_CIPHER_meth_set_iv_length	gost_crypt.c:118	Same as above.
29	EVP_CIPHER_meth_set_flags	gost_crypt.c:119	Same as above.
30	EVP_CIPHER_meth_set_init	gost_crypt.c:120	Same as above.
31	EVP_CIPHER_meth_set_do_cipher	gost_crypt.c:121	Same as above.
32	EVP_CIPHER_meth_set_cleanup	gost_crypt.c:122	Same as above.
33	EVP_CIPHER_meth_set_impl_ctx_size	gost_crypt.c:123	Same as above.
34	EVP_CIPHER_meth_set_set_asn1_params	gost_crypt.c:124	Same as above.
35	EVP_CIPHER_meth_set_get_asn1_params	gost_crypt.c:125	Same as above.
36	EVP_CIPHER_meth_set_ctrl	gost_crypt.c:126	Same as above.
37	EVP_CIPHER_meth_free	gost_crypt.c:127,137	Same as above.

---

EVP_MD_meth_* (removed)

№	Removed interface	Where found (file:lines)	Migration strategy
38	EVP_MD_meth_new	gost_md.c:53	Move to gost_md_legacy.c.
39	EVP_MD_meth_set_result_size	gost_md.c:54	Same as above.
40	EVP_MD_meth_set_input_blocksize	gost_md.c:55	Same as above.
41	EVP_MD_meth_set_app_datasize	gost_md.c:56	Same as above.
42	EVP_MD_meth_set_flags	gost_md.c:57	Same as above.
43	EVP_MD_meth_set_init	gost_md.c:58	Same as above.
44	EVP_MD_meth_set_update	gost_md.c:59	Same as above.
45	EVP_MD_meth_set_final	gost_md.c:60	Same as above.
46	EVP_MD_meth_set_copy	gost_md.c:61	Same as above.
47	EVP_MD_meth_set_cleanup	gost_md.c:62	Same as above.
48	EVP_MD_meth_set_ctrl	gost_md.c:63	Same as above.
49	EVP_MD_meth_free	gost_md.c:64,77	Same as above.

---

EVP_PKEY_meth_* (removed)

№	Removed interface	Where found (file:lines)	Migration strategy
50	EVP_PKEY_meth_new	gost_pmeth.c:1153	Move to gost_pmeth_legacy.c.
51	EVP_PKEY_meth_set_ctrl	gost_pmeth.c:1160–1251	Same as above.
52	EVP_PKEY_meth_set_sign	gost_pmeth.c:1162,1181,1201	Same as above.
53	EVP_PKEY_meth_set_verify	gost_pmeth.c:1163,1182,1202	Same as above.
54	EVP_PKEY_meth_set_keygen	gost_pmeth.c:1165–1255	Same as above.
55	EVP_PKEY_meth_set_encrypt	gost_pmeth.c:1167,1186,1206	Same as above.
56	EVP_PKEY_meth_set_decrypt	gost_pmeth.c:1170,1189,1209	Same as above.
57	EVP_PKEY_meth_set_derive	gost_pmeth.c:1171,1190,1210	Same as above.
58	EVP_PKEY_meth_set_paramgen	gost_pmeth.c:1173,1192,1212	Same as above.
59	EVP_PKEY_meth_set_check	gost_pmeth.c:1175,1195,1215	Same as above.
60	EVP_PKEY_meth_set_public_check	gost_pmeth.c:1176,1196,1216	Same as above.
61	EVP_PKEY_meth_set_signctx	gost_pmeth.c:1221–1253	Same as above.
62	EVP_PKEY_meth_set_init	gost_pmeth.c:1224–1263	Same as above.
63	EVP_PKEY_meth_set_cleanup	gost_pmeth.c:1225–1264	Same as above.
64	EVP_PKEY_meth_set_copy	gost_pmeth.c:1226–1266	Same as above.

---

ENGINE_* API (removed)

№	Removed interface	Where found (file:lines)	Migration strategy
65	ENGINE_new	gost_eng.c:540	Move to gost_eng_legacy.c.
66	ENGINE_add	gost_eng.c:542; gost_prov.c:65	Remove ENGINE dependency from provider code. Provider must depend only on internal interfaces and must not rely on the OpenSSL ENGINE API; ENGINE support is allowed only in legacy-only builds.
67	ENGINE_free	gost_eng.c:543	Same as above.
68	ENGINE_register_all_complete	gost_eng.c:492	Same as above.
69	ENGINE_register_ciphers	gost_eng.c:476	Same as above.
70	ENGINE_register_digests	gost_eng.c:477	Same as above.
71	ENGINE_set_ciphers	gost_eng.c:410	Same as above.
72	ENGINE_set_digests	gost_eng.c:406	Same as above.
73	ENGINE_set_id	gost_eng.c:394	Same as above.
74	ENGINE_set_name	gost_eng.c:398	Same as above.
75	ENGINE_set_ctrl_function	gost_eng.c:427	Same as above.
76	ENGINE_set_destroy_function	gost_eng.c:431	Same as above.
77	ENGINE_set_finish_function	gost_eng.c:433	Same as above.
78	ENGINE_set_init_function	gost_eng.c:432	Same as above.
79	ENGINE_set_cmd_defns	gost_eng.c:423	Same as above.

---

Common Build Rule for Legacy Code

All legacy modules must be compiled only when the legacy build flag is explicitly enabled:

-DGOST_ENABLE_LEGACY=1

When the flag is not set, legacy modules must not be compiled, and only modern provider-based implementations should be used.

Provider Architecture Constraint: EVP Recursion

While migrating away from deprecated ENGINE and EVP-based APIs, an important architectural constraint of the OpenSSL provider model must be explicitly addressed.

Background

Historically, the GOST implementation relied on the EVP interface internally:

• ENGINE-based implementations were allowed to use EVP_MD_CTX, EVP_CIPHER_CTX, etc.
• EVP acted both as a dispatcher and a helper layer
• Calling EVP functions from inside ENGINE code was valid and widely used

This design assumption no longer holds for providers.

Problem in Provider Context

In the provider model:

• EVP is a consumer of providers
• Provider dispatch functions are called by EVP
• A provider must not call EVP back

If a provider implementation uses EVP internally, the following cycle occurs:

1. Application calls EVP
2. EVP routes the call to the provider
3. Provider calls EVP internally
4. EVP routes the call back to the provider
5. Infinite recursion / undefined behavior

This makes any EVP-based implementation inside a provider invalid by design.

Consequence

Therefore:

• What was acceptable for ENGINE is not acceptable for PROVIDER
• Provider implementations must be EVP-independent
• All digest, MAC, and cipher logic inside the provider must operate on
  provider-native, internal contexts, not on EVP_*_CTX

Relation to Migration Tasks

This architectural requirement is the underlying reason for the following tasks:

• Refactor gost-engine cipher implementation to use internal cipher contexts #501–Refactor legacy MAC implementations to use internal OMAC context #503: introduce internal cipher, digest, and MAC interfaces
• Decouple cipher provider logic from EVP_CIPHER* APIs #504–Decouple digest and MAC provider logic from EVP_MD* APIs #505: refactor provider implementations to use provider-native contexts
• Remove ENGINE initialization from provider startup #506: ensure provider lifecycle is fully independent of ENGINE

Without this change, correct provider operation is impossible, regardless of
deprecated API cleanup.

Planned Changes

• Migrate CMAC_* to EVP_MAC interface #497
  (fixes: Deprecated MAC Methods §1–6)

• Migrate HMAC_* to EVP_MAC interface #498
  (fixes: Deprecated MAC Methods §7–12)

• Disable ENGINE-based tests unless explicitly enabled #499
  Ensure ENGINE-based tests are neither built nor executed by default.
  Legacy ENGINE tests must be opt-in and enabled only when the GOST_ENABLE_LEGACY build flag is explicitly set.
  (fixes: ENGINE_ API removed §65–79; legacy test coverage)*

• Provide a local replacement for deprecated EC_KEY_* APIs #500
  Implement a minimal internal EC_KEY layer used only when OpenSSL is built with OPENSSL_NO_DEPRECATED_3_0.
  This is a transitional solution until full migration to provider key management is feasible.
  (fixes: Deprecated EC_KEY Interfaces §13–21)

• Refactor gost-engine cipher implementation to use internal cipher contexts #501
  Introduce an internal cipher interface that depends only on internal cipher contexts
  (e.g. ossl_gost_cipher_ctx, gost_mgm_ctx) and does not depend on EVP_CIPHER or EVP_CIPHER_CTX.
  (fixes: Deprecated EVP_CIPHER_CTX Accessors §22–24; prerequisite for provider implementation depending only on internal interfaces and not on the OpenSSL ENGINE interface (§66))

• Refactor legacy digest implementations to use internal digest contexts #502
  Introduce an internal digest interface that depends only on internal digest contexts
  (e.g. ossl_gost_digest_ctx) and does not depend on EVP_MD or EVP_MD_CTX.
  (fixes: prerequisite for provider implementation depending only on internal interfaces and not on the OpenSSL ENGINE interface (§66))

• Refactor legacy MAC implementations to use internal OMAC context #503
  Introduce an internal MAC interface that depends only on an internal OMAC context (OMAC_CTX) and does not depend on EVP_MD, EVP_MD_CTX, or legacy EVP digest wrappers.
  (fixes: prerequisite for provider implementation depending only on internal interfaces and not on the OpenSSL ENGINE interface (§66))

• Decouple cipher provider logic from EVP_CIPHER* APIs #504
  Refactor provider cipher implementations to operate exclusively on internal cipher contexts and descriptors, without relying on EVP_CIPHER, EVP_CIPHER_CTX, or other legacy EVP wrappers.
  (fixes: cipher provider is no longer dependent on the ENGINE §66)

• Decouple digest and MAC provider logic from EVP_MD* APIs #505
  Refactor provider digest and MAC implementations to operate on provider-native contexts, without relying on EVP_MD, EVP_MD_CTX, or legacy EVP digest interfaces.
  (fixes: digest and mac provider is no longer dependent on the ENGINE §66)

• Remove ENGINE initialization from provider startup #506
  Ensure provider initialization is completely independent of ENGINE registration and lifecycle.
  (fixes: provider is no longer dependent on the ENGINE §66)

• Move all ENGINE-based code into legacy-only modules and compile them only when GOST_ENABLE_LEGACY=1 is enabled #507
  Move all ENGINE registration code, including EVP_CIPHER_meth_, EVP_MD_meth_, EVP_PKEY_meth_, and other legacy compatibility glue for EVP_meth* APIs, into legacy.c files compiled conditionally.
  *(fixes: EVP_CIPHER_meth removed §27–37; EVP_MD_meth_* removed §38–49; EVP_PKEY_meth_* removed §50–64; ENGINE_ API removed §65–79)*

• Move all code using EVP_PKEY_assign and EVP_PKEY_get0 into legacy-only modules #508
  Isolate all direct access to deprecated EVP_PKEY internals behind legacy compatibility layers compiled only when legacy support is enabled.
  (fixes: Deprecated EVP_PKEY Accessors §25–26)

• Enable GitHub CI builds against OpenSSL master #509
  Update GitHub Actions workflows to build and test against the OpenSSL master branch, including configurations with OPENSSL_NO_DEPRECATED_3_0.
  This step serves as final validation that all removed and deprecated APIs have been fully eliminated from default builds.
  (fixes: final validation of all removed/deprecated APIs)

Expected Outcome

After completing the above tasks:

• Provider code is fully independent of deprecated ENGINE and EVP interfaces
• ENGINE support is clearly marked as deprecated and optional
• The project builds and tests successfully with OpenSSL configurations that disable deprecated APIs
• The codebase is aligned with OpenSSL’s long-term provider-first architecture