Scan fails with error 'File can't be indexed twice'

[liortal53]

Description

When attempting to scan my Unity3d project (c# usig Mono), scanning fails with this error:

ERROR: Error during SonarQube Scanner execution
ERROR: File Assets/DynamicContent/Events/Jackpot/@grobold_golden_jackpot.fnt can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files

Please provide a succinct description of your issue.

The Unity game engine auto-generates a .sln and .csproj files for the code in the project. Since the project contains art assets, the .csproj files also contain references to these files, for example (taken from the attached .csproj file):

 <ItemGroup>
    <None Include="Assets\DynamicContent\Events\Jackpot\@grobold_golden_jackpot.fnt" />
    <None Include="Assets\DynamicContent\Events\Jackpot\@jackpot_simple_numbers.fnt" />
 </ItemGroup>

The problem is, that multiple .csproj files contain these source files, even though they are marked as "None" (e.g: not compiled).

Repro steps

Attaching the problematic .csproj files as well as the the scan log + matching .sonarqube folder.
Hopefully, this should suffice for diagnosing the issue. If not, let me know.

• SonarC# Version: 6.6 (build 3969)
• Visual Studio Version - 2017 community edition
• If running through the Scanner for MSBuild, its version - 3.0.2

csharp_scan_bug.zip

[Evangelink]

Relates (duplicates?) #838

[liortal53]

Not sure. This is not the same error

…

On Nov 20, 2017 11:34, "Amaury Levé" ***@***.***> wrote: Relates (duplicates?) #838 <#838> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxVl881ONREIZQ3kOj-4VsmUdrhvfFwks5s4UezgaJpZM4QjhDH> .

[Evangelink]

Hi @liortal53,

Do you still have this issue with the RC version of the scanner for msbuild and SonarC#?

[dbmeneses]

This is related to https://jira.sonarsource.com/browse/SONARMSBRU-336

[liortal53]

yes, i still see this issue. I attached a sample .sln and .csproj files that result in this issue. The problem is that with Unity, the .csproj files are auto-generated by the Unity editor. It adds some of these files (which are duplicates) to different .csproj files for some reason, even though they are added with a <None> action so they are not really processed by the compilation process. Maybe the scanner should ignore these files and not count them as duplicates?

…

On Mon, Nov 27, 2017 at 11:02 AM, Amaury Levé ***@***.***> wrote: Hi @liortal53 <https://github.com/liortal53>, Do you still have this issue with the RC version of the scanner for msbuild and SonarC#? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxVl8KEUI51Dy5YeTZmw8YZGR5kOL0mks5s6nqggaJpZM4QjhDH> .

[Evangelink]

Hi @liortal53,

As @dbmeneses was suggesting we think we found your error and we will try to have it fixed for the next release. Please stay tuned.

[liortal53]

awesome, looking forward to that. what is the release frequency? are you releasing frequently (e.g: once in a few weeks) ?

…

On Mon, Nov 27, 2017 at 3:26 PM, Amaury Levé ***@***.***> wrote: Hi @liortal53 <https://github.com/liortal53>, As @dbmeneses <https://github.com/dbmeneses> was suggesting we think we found your error and we will try to have it fixed for the next release. Please stay tuned. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxVl2YDr9fzDWCO5KSoQm7bieqULXl6ks5s6rhngaJpZM4QjhDH> .

[Evangelink]

We try (I insist on this) to release every 2 or 3 weeks. But sometime we have to give a bit more focus on some of the tools and so we kind of delay the others. For example we are a bit late of VSTS extension (regarding .Net Core support) and so we will focus on this subject on the next sprint.

[dbmeneses]

Here is the ticket: https://jira.sonarsource.com/browse/SONARMSBRU-354

[pmaslan]

I have similar issue with .NET Core application. All our .NET Core analysis failed with error: "File * can't be indexed twice. Please check that inclusion/exclusion
patterns produce disjoint sets for main and test files.". It occurs for all cs files (if I exclude one it occurs on next one). Each cs file exists only once and is referenced once.

Issue happend after update SonarQube from 6.2 to 6.7.

SonarC# Version: 6.6
Scanner for MSBuild: 3.0.2.656; 3.1-beta.1
MSBuild: 15.4.8.50001

[dbmeneses]

@pmaslan could you please provide the full logs? The problem reported here is due to a quite particular project structure.

[pmaslan]

Yes, I can.

Project structure is:
Alerts
|- Alerts.Business
|- Alerts.Business.Tests
|-Alerts.Persistence
|- Alerts.Tests

Script for analysis looks like this:

 C:\sonar-scanner\sonar-scanner-3.0.3.778\..\SonarQube.Scanner.MSBuild.exe begin /d:sonar.sources="C:\bamboo-agent-home\xml-data\build-dir\SHT-AL0-SQ" /d:sonar.exclusions=**/*Test*/**,**/scripts/**,**/dotCover/**,**/InstallScripts/*,**/obj/**,**/bin/** /d:sonar.sourceEncoding=UTF-8 /d:sonar.language=cs /d:sonar.scm.disabled=true /key:"AL" /name:"Alerts" /version:"1.0.54

MSBuild /t:Rebuild C:\bamboo-agent-home\xml-data\build-dir\SHT-AL0-SQ\Alerts.sln

C:\sonar-scanner\sonar-scanner-3.0.3.778\..\SonarQube.Scanner.MSBuild.exe end

sonar-project.zip
plan-131105361-SQ-54.log

[dbmeneses]

@pmaslan I see that the root module has all files included in its source set:

INFO: Source paths: .

This is not expected. Are you overriding some of the scanner properties?

[pmaslan]

I moved all sources to src directory and run it again (with sonar.sources=. and then with .\src). Nothing changed.

Overrided properties:
/d:sonar.sources="C:\bamboo-agent-home\xml-data\build-dir\SHT-AL0-SQ" /d:sonar.exclusions=**/*Test*/**,**/scripts/**,**/dotCover/**,**/InstallScripts/*,**/obj/**,**/bin/** /d:sonar.sourceEncoding=UTF-8 /d:sonar.language=cs /d:sonar.scm.disabled=true /key:"AL" /name:"Alerts" /version:"1.0.61"

plan-131105361-SQ-61.log

[dbmeneses]

Could you try to run the analysis without overriding 'sonar.sources'? The Scanner for MSBuild should automatically pickup all your sources based on the files that are in your solution.

[pmaslan]

I removed sonar.source and analysis was success.

@dbmeneses Thank you very much for your help.

[Evangelink]

Hi @liortal53, I am closing this ticket and will be working on the one on JIRA as it is linked to the scanner product.

[equiman]

The problem it's caused by /d:sonar.source="Project" or /d:sonar.tests="Project.Tests", just remove and works.

[MG12Coder]

Hi,
I am getting the same error while running the .net codebase build from teamcity.
[09:14:31][Step 2/2] Caused by: File [moduleKey=Bmpe_BmpeUiBaseOnlyKsorItem, relative=BMPECommunicationTest/App.xaml.cs, basedir=C:\BuildAgent\work\BWKsor] can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files

I have setup two build steps for execution 1) msbuild 2) sonar runner

These are the project sonar setting on my teamcity agent server
#edit these values accordingly
sonar.projectKey=com.rbs.mib:dotnet.BMPE
sonar.projectVersion=1.0
sonar.projectName=BMPE-BaseOnly-Ui
sonar.dotnet.visualstudio.solution.file=BMPE-BaseOnly.sln
sonar.dotnet.visualstudio.testProjectPattern=.Tests;.UnitTests
#end edit values

sonar.language=cs
sonar.sourceEncoding=UTF-8

sonar.ndeps.mode=skip
sonar.fxcop.mode=skip
sonar.stylecop.mode=skip
sonar.scm.enabled=true
sonar.skippedModules=BMPE.Tests,CommonControls,BMPE.DTO

and on teamcity where i have setup two build setup like below

can you please help me in this matter
what needs to be changed and where?

[Evangelink]

Hi @MG12Coder,

Disclaimer: I am not familiar with the Team City plugin so my answer might not be totally accurate.

I see that you are providing the Sources location: property. As you might see from previous answers on this thread we recommend not to manually specify sources as the scanner will automatically detect them from the projects in your solution.

Could you give a try by blanking this property?

[MG12Coder]

Hi Evangelink,

i tried removing source location property but does not help

[Evangelink]

@MG12Coder,

Could you please open a new GitHub issue and post the verbose logs (/d:sonar.verbose=true) of the end step?

[the-nose-knows]

@Evangelink can you please see about adding a console message for setting /d:sonar.verbose=true for the MSBuild wrapper? By default, it says to use -X which doesn't seem to work unless I'm not using the MSBuild wrapper, causing SQ to fail due to unknown command-line arguments.

Here's an example of the console-output from one of my Jenkins builds:

15:18:55 SonarQube Scanner for MSBuild 4.0.2
15:18:55 Default properties file was found at C:\SonarQube\MSBuild\sonar-scanner-msbuild-4.0.2.892\SonarQube.Analysis.xml
15:18:55 Loading analysis properties from C:\SonarQube\MSBuild\sonar-scanner-msbuild-4.0.2.892\SonarQube.Analysis.xml
15:18:55 Pre-processing started.
15:18:55 Preparing working directories...
15:18:55 15:18:55.798 15:18:55.794 Unrecognized command line argument: -X

Apologies if you're not the go-to person; I've just been trying to debug some issues with the MSBuild Wrapper today, and been wondering how to enable verbose output.

[Evangelink]

Hi @the-nose-knows,

Could you please create a separate ticket on the Scanner for MSBuild repository here? And could you add the log where it is asked to use the -x command?

Thank you

[anikredd]

[11:20:25] 18:20:25.092 INFO: ------------------------------------------------------------------------

[11:20:25] 18:20:25.092 ERROR: Error during SonarQube Scanner execution

[11:20:25] File [moduleKey=27158:com.att.floodportal:threat_map_angular, relative=client/src/app/components/rap.component.html, basedir=/opt/app/m61407_node/sdt-eco.vci.att.com_19592/workspace/l_threat_map_angular_master-XE7WZQW2ZBU5B6GKL5CB7CXENJT6UUZEWBWIVCLY7ZKLBTRF7ZWA] can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files

[11:20:25] Return code: 1.

/opt/app/m61407_node/sdt-eco.vci.att.com_19592/workspace/l_threat_map_angular_master-XE7WZQW2ZBU5B6GKL5CB7CXENJT6UUZEWBWIVCLY7ZKLBTRF7ZWA/util/node_modules/gulp-sonar/index.js:74
throw new PluginError('gulp-sonar', format('Return code: %d.', code));
^
Error: Return code: 1.

==============================================

projectKey: **** ,
projectName: **** ,
exclusions:'client/src/app/components//*.spec.ts,util//,client/node_modules/**/,client/src/static/,client/src/static/**/,client/dist/,client/dist/**/,client/coverage/,client/coverage/**/',
projectVersion: '1.0.0',
sources: 'client',
tests: 'client/src/app/components'',
language: 'js',
sourceEncoding: 'UTF-8',
javascript: {
lcov: {
reportPath: 'client/coverage/lcov.info'
}
},

This is the Problem am facing

[chilukuridevbuzz]

Me too facing same issue

[sagar221]

Hi, Is there a way to control Sonar Indexing to Index only specified language or extension. I have used sonar.language=cs, but while performing analysis sonar index identifies unspecified language/file extensions and later it performs analysis only on specified language making other files as skipped.

[kiransterling]

I am facing the same issue . Let me know what is the resolution.

[andrei-epure-sonarsource]

To give some general advice on this problem:

• regarding indexing files, the src and the test set of files must be disjoint sets, so please verify your configuration (for example, @anikredd configured above the tests as a subfolder of src folder, therefore the sets are not disjoint) - see related SO answer
• you can also read the Narrowing the Focus documentation on how to exclude files in order to achieve disjoint sets (i.e. by using sonar.exclusions and/or sonar.test.exclusions)

@sagar221 @kiransterling - this is the sonar-dotnet repository which contains just the code for the language analyzers (SonarCSharp and SonarVbnet Roslyn analyzers). If you have problems with the scanner for MSBuild, please open a new issue:

• either on our online community forum (https://community.sonarsource.com/)
• or on the Sonar Scanner for MSBuild repo (https://github.com/SonarSource/sonar-scanner-msbuild)

I will be locking the conversation on this thread, as it does not contain actionable requests on the sonar-dotnet codebase.

Thank you for your understanding 🙏.