fix return code

[ctron]

Fix #2512

Notes:

• Pull requests cannot be accepted until the PR follows the contributing guidelines. In particular, each commit must have at least one Signed-off-by: line from the committer to certify that the contribution is made under the terms of the Developer Certificate of Origin.
• This is just a template, so feel free to use/remove the unnecessary things

Description

A few sentences describing the overall goals of the pull request's commits.

In all other cases, the invalid public key error, caused by an ASN1 length mismatch error, is calculated as:

https://github.com/ARMmbed/mbedtls/blob/5cb54f7b27e1ebc1602233f50c44c1ff05304852/library/pkparse.c#L638-L640

In the case this PR fixes, a + is missing:

https://github.com/ARMmbed/mbedtls/blob/5cb54f7b27e1ebc1602233f50c44c1ff05304852/library/pkparse.c#L664-L666

From a mathematical point of view, it doesn't matter, since in one case it expands as:

-0x3B00 + -0x0066

And in the other as:

-0x3B00 -0x0066

Which results in the same outcome.

However, should someone decide to change to defined values in the future, it might break compilation or come to a different outcome.

Status

READY

Requires Backporting

Yes, #3707 and #3708.

Additional comments

Any additional information that could be of interest

Todos

• Tests
• Documentation
• Changelog updated
• Backported

Steps to test or reproduce

Outline the steps to test or reproduce the PR here.

[ronald-cron-arm]

It is indeed better like this. Thanks. The CI is ok, the Mbed OS Testing is currently broken and we are about to remove it. The error reported by "continuous-integration/jenkins/pr-head" and "PR-3705-head TLS Testing" are CI glitches.

[gilles-peskine-arm]

Thanks! This was actually reported before #2512 but as there was no functional issue we'd never bothered to make the patch.

[ronald-cron-arm]

@ctron I've done the backport PRs: #3707 and #3708 to fix it everywhere while at it. I hope this is ok with you.