fix: Fix handling of large json data when writing to file via --json-file-output [CLI-46]

[j-luong]

Pull Request Submission

Please check the boxes once done.

The pull request must:

• Reviewer Documentation
  • follow CONTRIBUTING rules
  • be accompanied by a detailed description of the changes
  • contain a risk assessment of the change (Low | Medium | High) with regards to breaking existing functionality. A change e.g. of an underlying language plugin can completely break the functionality for that language, but appearing as only a version change in the dependencies.
  • highlight breaking API if applicable
  • contain a link to the automatic tests that cover the updated functionality.
  • contain testing instructions in case that the reviewer wants to manual verify as well, to add to the manual testing done by the author.
  • link to the link to the PR for the User-facing documentation
• User facing Documentation
  • update any relevant documentation in gitbook by submitting a gitbook PR, and including the PR link here
  • ensure that the message of the final single commit is descriptive and prefixed with either feat: or fix: , others might be used in rare occasions as well, if there is no need to document the changes in the release notes. The changes or fixes should be described in detail in the commit message for the changelog & release notes.
• Testing
  • Changes, removals and additions to functionality must be covered by acceptance / integration tests or smoke tests - either already existing ones, or new ones, created by the author of the PR.

Pull Request Review

All pull requests must undergo a thorough review process before being merged.
The review process of the code PR should include code review, testing, and any necessary feedback or revisions.
Pull request reviews of functionality developed in other teams only review the given documentation and test reports.

Manual testing will not be performed by the reviewing team, and is the responsibility of the author of the PR.

For Node projects: It’s important to make sure changes in package.json are also affecting package-lock.json correctly.

If a dependency is not necessary, don’t add it.

When adding a new package as a dependency, make sure that the change is absolutely necessary. We would like to refrain from adding new dependencies when possible.
Documentation PRs in gitbook are reviewed by Snyk's content team. They will also advise on the best phrasing and structuring if needed.

Pull Request Approval

Once a pull request has been reviewed and all necessary revisions have been made, it is approved for merging into
the main codebase. The merging of the code PR is performed by the code owners, the merging of the documentation PR
by our content writers.

What does this PR do?

This PR fixes an error occurring when using --json-file-output when writing large JSON objects into the specified file. The error surfaced as a exception with a stack trace similar to the one below

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at Object.jsonStringifyLargeObject (/snapshot/project/dist/cli/webpack:/snyk/src/lib/json.ts:17:16)
    at Object.extractDataToSendFromResults (/snapshot/project/dist/cli/webpack:/snyk/src/lib/formatters/test/format-test-results.ts:105:27)

The root cause is a size limitation of Nodejs V8 string size limit when converting objects to strings using JSON.stringify in json.ts.

The solution introduced in this PR uses a streaming mechanism to work around this limitation.

In order to minimise risk, the new streaming implementation is used only when:

• the originial JSON.stringify fails in json.ts.
• only for --json-file-output

The existing behaviour when JSON.stringify fails is to throw a RangeError error, and occurs when using the --json and --json-file-output flags.

This PR should also provide a better error message, instead of throwing a RangeError, when JSON.stringify fails with the --json flag.

For the above reasons, the risk is assumed to be LOW

Where should the reviewer start?

Looking at the acceptance test test --json-file-ouput handles responses larger than 512Mb string size limit in v8 to get an idea of what causes the issue.

How should this be manually tested?

It can be tested via building a local version of the CLI binary:

• checkout branch
• make clean build
• run local binary ./binary-releases/<local-binary> container test --json-file-output=./output.json <image:tag>

It can also be quickly tested by running the legacy CLI as the changes are there:

• checkout branch
• from project root run: npx ts-node ./src/cli/index.ts container test --json-file-output=./output.json <image:tag>

Manually testing the fix will be difficult without manipulating the JSON object in code, as it depends on the image having an extremely high number of issues. The acceptance test test --json-file-ouput handles responses larger than 512Mb string size limit in v8 shows how to do this if desired.

Any background context you want to provide?

What are the relevant tickets?

Screenshots

Additional questions

[github-actions]

	Warnings
⚠️	Since the CLI is unifying on a standard and improved tooling, we're starting to migrate old-style imports and exports to ES6 ones. A file you've modified is using either module.exports or require(). If you can, please update them to ES6 import syntax and export syntax. Files found: src/cli/commands/test/index.ts src/cli/main.ts src/lib/json.ts test/jest/unit/json-file-output.spec.ts
⚠️	"fix: stop heap out of memory issues from making massive JSON objects in tests" is too long. Keep the first line of your commit message under 72 characters.

Generated by 🚫 dangerJS against 5448caf

[thisislawatts]

nitpick (non-blocking): Can we align on using imports rather than requires? Marking this as non-blocking as I ideally this is something picked up by tooling. Likely a bunch of other places across the code base this needs to be resolved.

[j-luong]

This is actually a linting rule specifically for lodash. Not sure why it's there, but didn't want to change it in this PR.