Add bounds checks for size_t to int truncation in RSA_METHOD calls

[justsmth]

Description of changes:

The RSA_METHOD function pointers (sign_raw, verify_raw, encrypt,
decrypt) accept int max_out parameters for OpenSSL API compatibility,
but the calling functions receive max_out as size_t.

In practice, max_out is bounded by the RSA modulus size and would never
approach INT_MAX. These checks are a defense-in-depth measure. The error
code (ERR_R_OVERFLOW) and error-handling pattern (*out_len = 0; return 0)
are consistent with existing overflow checks in the codebase.

Testing:

Existing RSA tests cover the normal code paths. The new bounds checks
only trigger for max_out values exceeding INT_MAX, which cannot occur
with valid RSA key sizes.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 25.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.13%. Comparing base (2f1bbe4) to head (619c5e4).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crypto/rsa_extra/rsa_crypt.c	25.00%	6 Missing ⚠️
crypto/fipsmodule/rsa/rsa.c	25.00%	3 Missing ⚠️
crypto/fipsmodule/rsa/rsa_impl.c	25.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3084      +/-   ##
==========================================
+ Coverage   78.11%   78.13%   +0.01%     
==========================================
  Files         689      689              
  Lines      121596   121612      +16     
  Branches    16981    16986       +5     
==========================================
+ Hits        94981    95016      +35     
+ Misses      25729    25714      -15     
+ Partials      886      882       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.