output: fix logic error

[ProjectMutilation]

The logical error may have been made here. Comparison with the upper
bound of the variable type does not make sense. It may be worth adding
the cast of one of the multiplication operands to the 64-bit type for
avoiding overflow

Found by Security Code with Svace static analyzer
Bug: #5789

Signed-off-by: Maxim Korotkov m.korotkov@securitycode.ru

Make sure these boxes are signed before submitting your Pull Request -- thank you.

• I have read the contributing guide lines at https://suricata.readthedocs.io/en/latest/devguide/codebase/contributing/contribution-process.html
• I have signed the Open Information Security Foundation contribution agreement at https://suricata.io/about/contribution-agreement/
• I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)

Link to redmine ticket: https://redmine.openinfosecfoundation.org/issues/5789

Describe changes:

• Added cast to ULL one of the operands of multiplication for avoiding overflow

#suricata-verify-pr:
#suricata-verify-repo:
#suricata-verify-branch:
#suricata-update-pr:
#suricata-update-repo:
#suricata-update-branch:
#libhtp-pr:
#libhtp-repo:
#libhtp-branch:

[ProjectMutilation]

@victorjulien hi, can you explain why tests were failing please? I don't have idea

[victorjulien]

Probably needs a rebase to master. The failing test was merged yesterday following a fix in master.

[codecov]

Codecov Report

Merging #8530 (43014bf) into master (4ad5e04) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

❗ Current head 43014bf differs from pull request most recent head 2ec95f0. Consider uploading reports for the commit 2ec95f0 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8530      +/-   ##
==========================================
- Coverage   81.95%   81.90%   -0.05%     
==========================================
  Files         967      967              
  Lines      278373   278373              
==========================================
- Hits       228129   228001     -128     
- Misses      50244    50372     +128     

Flag	Coverage Δ
fuzzcorpus	64.05% <100.00%> (+0.03%)	⬆️
suricata-verify	59.80% <100.00%> (-0.11%)	⬇️
unittests	63.36% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jufajardini]

Tests pass now :)
Could you please square these commits? That rebase commit isn't something we keep in our commit history, like that.

[jufajardini]

Please see my inline comment :)

[ProjectMutilation]

Please see my inline comment :)

yep, i saw it, I'll do it a little later

[ProjectMutilation]

i'll create new pr with clean history :)

[jufajardini]

Followed by: #8547