How to suppress the error "AnalyzerResult contains packages that are also projects."

[mawl]

Hey,

How can we suppress following error concerning npm dependencies in a monorepo? Some of the dependencies are also projects in it.

07:18:50.479 [DefaultDispatcher-worker-4] ERROR org.ossreviewtoolkit.analyzer.AnalyzerResultBuilder - Multiple projects with the same id 'Yarn2::client:0.1.0' found. 
Not adding the project defined in 'https://same-url/package.json' to the analyzer results as it duplicates the project defined in 'https://same-url/package.json'.
Exception in thread "main" java.lang.IllegalArgumentException: AnalyzerResult contains packages that are also projects.
Duplicates: '[Identifier(type=NPM, namespace=@foo, name=some-library, version=0.1.15)]'.
	at org.ossreviewtoolkit.analyzer.AnalyzerResultBuilder.build(AnalyzerResultBuilder.kt:49)
	at org.ossreviewtoolkit.analyzer.AnalyzerState.buildResult(Analyzer.kt:250)
	at org.ossreviewtoolkit.analyzer.Analyzer.analyzeInParallel(Analyzer.kt:169)
	at org.ossreviewtoolkit.analyzer.Analyzer.analyze(Analyzer.kt:116)
	at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:284)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:81)
Evaluate

ort/analyzer/src/main/kotlin/AnalyzerResultBuilder.kt

Line 53 in 30ddfed

"AnalyzerResult contains packages that are also projects. Duplicates: '$duplicateIds'."

[nicorikken]

I ran into the same issue unintentionally as the conflict sees to come from a directory I excluded, so I'm interested in other ways to work around this error.

Perhaps in the monorepo setup you can scan all projects (subdirectories) separately to avoid the issue of duplicate dependencies?

Edit: I got an answer through chat for my situation by @sschuberth that clarified the inner workings for me regarding excludes, so that isn't an option in this case.

As usual (in ORT), excludes don't actually prevent the analysis. ORT still tries to run the analysis and only marks the results of the analysis as excluded. Which means the consistency check you're running into is still executed.

[mnonnenmacher]

This check was introduced in df84062 because if the analyzer result contains projects and packages with the same identifier this can lead to unexpected behavior in later steps (one main issue was that the web app report was showing incorrect data). It usually indicates an issue in the package manager implementation which did not correctly identify a project dependency.

@mawl @nicorikken Please provide example projects that reproduce this issue so that we can identify the root cause.

[mawl]

Here is an example project reproducing this issue.

package-project.zip

We had to use a local package approach for that as we usually publish our packages to our private registry.

[mawl]

@mnonnenmacher: I have provided an example. Do you have any news for us concerning the progress? Thanks.

[mawl]

@mnonnenmacher, @sschuberth: can you please check the example? We have a few monorepos in our company where we would like to have ORT to work for, too. Have a good New Year :)

[sschuberth]

I haven't checked, but my hunch is that this happens because the monorepo contains both the source code to a package (which ORT treats as a project then), and another project is consuming the (binary) artifact for that package as a dependency (so here ORT treats it as a package). However, the id is the same in both cases. And that's something ORT does not currently support: Refer to the same id once as a project and once as a package / dependency within the same analyzer run.

[sschuberth]

BTW, as of #6241 you should actually also get a better error message in this case.

[mawl]

During scan phase it is possible to exclude PROJECTs from scanning. Wouldn't it be possible to implement the same option for the analyze phase as a workaround, like not treating them?

Usage: ort scan [OPTIONS]
  Run external license / copyright scanners.
Input Options:
...
  --package-types [PACKAGE|PROJECT]
                                   A comma-separated list of the package types
                                   from the ORT file's analyzer result to
                                   limit scans to. (default: [PACKAGE,
                                   PROJECT])
...

[sschuberth]

Wouldn't it be possible to implement the same option for the analyze phase as a workaround, like not treating them?

See #5968 😉

[sschuberth]

Wouldn't it be possible to implement the same option for the analyze phase as a workaround, like not treating them?

See #5968 😉

The PR has been merged, which answers the question on how to suppress the error by skipping paths with duplicate project / package ids.