Improve logging

[ghost]

• Change the "received request" log level
• Better define the default logging fields
• Remove useless logs
• Log in case of error in the mongoclientinterface, as InternalError hides all the underlying information
• Pass the right error in case of error during deletion, to actually fix a bug in case of mongodb erroor leading to no error log AND deletion of data from the storage layer, but not in mongodb
• In the dataWrapper some logs were in trace, meaning no info available in case of error when handling data, and harder debugging
• Adding multiple tests missing today, affecting code coverage due to the logging updates
• Updated the in memory data store, as we need the head method to properly unit test backbeat routes.

Issue: ARSN-471

[bert-e]

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue ARSN-471 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 8.1.148

• 8.2.4

Please check the Fix Version/s of ARSN-471, or the target
branch of this pull request.

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 5 lines in your changes missing coverage. Please review.

Project coverage is 68.35%. Comparing base (e245687) to head (5c61382).
Report is 7 commits behind head on development/8.1.

Files with missing lines	Patch %	Lines
...orage/metadata/mongoclient/MongoClientInterface.ts	60.00%	2 Missing ⚠️
lib/auth/v2/headerAuthCheck.ts	0.00%	1 Missing ⚠️
lib/s3routes/routes.ts	87.50%	1 Missing ⚠️
lib/storage/data/DataWrapper.js	75.00%	1 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           development/8.1    #2300      +/-   ##
===================================================
+ Coverage            66.42%   68.35%   +1.92%     
===================================================
  Files                  216      216              
  Lines                17443    17452       +9     
  Branches              3560     3566       +6     
===================================================
+ Hits                 11587    11929     +342     
+ Misses                5840     5507     -333     
  Partials                16       16              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[francoisferrand]

as a side note / complementary effort : we are changing some log levels here, but we should remember the logs (esp. debug/trace) are used in 2 contexts:

• Either when the user activate more verbose logging
  → display all the time, and indeed we may need to streamline these
• In case of error, every log related to the request is dumped again
  → in that specific context, having more (and separate lines) logs may/could be acceptable;
  → yet there may be an additional improvement in werelogs, to ensure these logs (some of which may already have been logged!) are clearly isolated: maybe by including them in the "final" json line or adding some sort of prefix/..., instead of just printing them like regular logs

That is probably a separate improvement, but sharing as I think it may bring a different point of view on the updating of logs.

[ghost]

→ yet there may be an additional improvement in werelogs, to ensure these logs (some of which may already have been logged!) are clearly isolated: maybe by including them in the "final" json line or adding some sort of prefix/..., instead of just printing them like regular logs

Indeed today we might log twice the same logs, in particular those with logging level >= to the log level, if a dump occurs.

Note: there is also a performance impact of having logs under the levels we listen to, because we are still processing them "in case of error", and that might lead to a lot of object copy/parsing, even if not useful.

[bert-e]

Incorrect fix version

The Fix Version/s in issue ARSN-471 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 8.1.148

• 8.2.5

Please check the Fix Version/s of ARSN-471, or the target
branch of this pull request.

[ghost]

Removing the last commit as we need to merge #2286 before releasing.

[bert-e]

Incorrect fix version

The Fix Version/s in issue ARSN-471 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 8.1.148

• 8.2.5

Please check the Fix Version/s of ARSN-471, or the target
branch of this pull request.

The following options are set: approve

[ghost]

/create_integration_branches

[bert-e]

Integration data created

I have created the integration data for the additional destination branches.

• this pull request will merge improvement/ARSN-471 into
  development/8.1
• w/8.2/improvement/ARSN-471 will be merged into development/8.2

The following branches will NOT be impacted:

• development/6.4
• development/7.10
• development/7.4
• development/7.70

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

The following options are set: create_integration_branches

[ghost]

/approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/8.1

• ✔️ development/8.2

The following branches have NOT changed:

• development/6.4
• development/7.10
• development/7.4
• development/7.70

Please check the status of the associated issue ARSN-471.

Goodbye williamlardier.

The following options are set: approve, create_integration_branches