Visual Studio Test Task unable to find custom logger

[aggarwalpulkit]

Required Information

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Visual Studio Test Task v2

Environment

• Server - Azure Pipelines or TFS on-premises?

  • If using TFS on-premises, provide the version: TFS 2019 on-premises
  • Agent - Hosted or Private: Private

-Machine: Windows Machine

Issue Description

When running Visual Studio Test Task with custom logger specified in runsettings file, visual studio test task is unable to find the logger, however running the same using dotnet vstest with same runsettings file works fine.

On further analysis and troubleshooting, it happens when using multi-agent run, single agent run works fine

Error logs

[Insert error from the logs here for a quick overview]

[ShreyasRmsft]

@aggarwalpulkit would it be possible to provide us with a sample repro? You can create a public GitHub repo with the code and a yaml file for the pipeline definition.

Also provide full zip logs with system.debug = true

And where are you placing your custom logger and how are you providing the path to it?

[BilalKhan199]

I'm facing same issue as well,
Custom logger does not work from Test Explorer , where the runsettings are

<?xml version="1.0" encoding="UTF-8"?>
<RunSettings>
  <RunConfiguration>
    <TestAdaptersPaths>.</TestAdaptersPaths>
  </RunConfiguration>>
  <LoggerRunSettings>
    <Loggers>
      <Logger friendlyName="ReportPortal">
        <Configuration>
          <Launch.Description>Ran from Visual Studio Test Explorer</Launch.Description>
        </Configuration>
      </Logger>
    </Loggers>
  </LoggerRunSettings>
</RunSettings>

Also in the Visual Studio Test Task v2 I'm specifing the logger in

does not work, seems it is not overwriting the logger at runtime.

Where same logger work fine with Visual Studio Test Task v1 or from dotnet vstest or when Parallelism is set to none

[aggarwalpulkit]

@ShreyasRmsft I won't be able to provide the test repo since I don't think its repo or logger related. I have tried number of tweaks on this, tried on both build and release pipelines and it all boils down to this setting on the agent job that causes it to break. If I use "Multi-Agent" then it is unable to find the custom logger, if I remove parallelism then it works fine.

[aggarwalpulkit]

Just to give a little more perspective and if someone is looking for a quick workaround like I was, I manually distributed my pipeline into multiple agent jobs, and divided the test cases between them using testcasefilter so I can mimic the behavior of using multiple agents and running tests in parallel.

[ShreyasRmsft]

@BilalKhan199 the 'OtherConsoleOption' input is not honored in the multi agent flow and there is a warning printed in the logs that calls out the same.

@aggarwalpulkit I never insinuated anywhere that it was repo or logger specific, that being said specifying a custom logger in the runsettings should work, the reason I ask for a sample repro and/or full logs with system.debug is to check where the custom logger is being placed and also how the path to the logger is being specified. The logs will tell us all we need to know and also anything else that might be interesting that hasn't yet occurred to me to ask at this point and will save a couple of days of to and fro asking for details.

The bare minimum we expect for any issue logged here is the full logs with system.debug = true.

[aggarwalpulkit]

@ShreyasRmsft I'll try to get you the logs tomorrow. Just if you want to look till then, the custom logger dll is in same folder as test dll in system.workingdirectory and is being specified using runsettings file. It is a parameterized logger implementing ITestLoggerWithParameters interface. Further logger dll is actually installed as a nuger on the project so it ends up in output directory. The runsettings file is also in same folder.

[ShreyasRmsft]

The runsettings might be the issue as we create a temp runsettings file in the agent temp folder with whatever ui to settings option manipulations the user has selected. Paths relative to runsettings will no longer be valid.

[aggarwalpulkit]

Interesting. So, what could be the possible fix? Since other console options is not supported in multi-agent pipeline and I did try specifying a testadapter path, and that didn't help either.
Also, I specify the runsettings file using the settings file option on the task and in the logs I do see that it is being read.

[ShreyasRmsft]

have you set system.debug to true? Please provide the logs i'll take a look.

[aggarwalpulkit]

customlogger_logs.zip
@ShreyasRmsft here you go for the logs. Please let me know if I can help provide anything else. Thanks!

[hvinett]

@aggarwalpulkit ,
I have seen two files
F:\vsts-default-1_work\1823\a\drop\FrameworkTest.CustomLogger.dll (file)
F:\vsts-default-1_work\1823\a\drop\VSTSIssue.Testlogger.dll (file)
In these which is test dll and which is logger dll?

[aggarwalpulkit]

@hvinett FrameworkTest.CustomLogger.dll is the dll that contains the testmethod. And, VSTSIssue.Testlogger.dll is the dll that contains the custom logger that implements ITestLoggerWithParameters.

Just a side comment for anyone else looking at this in future, I learnt it hard way by scanning thru every line of the logs that the custom logger dll needs to end with the words "testlogger" for it to be recognized, even if its specified in testadapterpath etc. It blows my mind, but it is what it is.

[Shakevg]

@aggarwalpulkit Try to specify Path to Test Adapters in runsettings file. It should help (as here )

<RunConfiguration>
	<TestAdaptersPaths>.</TestAdaptersPaths>
</RunConfiguration>

[aggarwalpulkit]

@Shakevg I think I mentioned it earlier, but I did try that and that doesn't really change anything.

[hvinett]

@aggarwalpulkit ,
Can you try moving your custom logger to extensions folder of testplatform which is getting picked up by agent?

[aggarwalpulkit]

@hvinett thats not a viable solution for us since the build machine is shared amongst a lot of apps, so installing something in a general location is not allowed.

[hvinett]

@aggarwalpulkit,
Can you attach the .csproj file of your custom logger project?
Also can you run it with vstest.console.exe and check whether the custom logger is getting initialized or not?

[aggarwalpulkit]

testproject.zip
@hvinett here you go! the zip contains both the test project and the custom logger code.

[hvinett]

@aggarwalpulkit ,

VSTSIssueTestLogger.txt

Can you try with this above csproj file?
Also specify the full path of net451 dll If you give custom logger path via test adapter path.
Let me know if the problem got resolved.
Also can you run with vstest.console.exe to make sure that specifying adapter path in vstest task is not an issue .

[ShreyasRmsft]

Closing due to no response. Please re-open with request information.

[gajendra2001]

@aggarwalpulkit @hvinett @ShreyasRmsft I also wanted to add logger in multiagent test run I am also facing the same issue. Even after custom logger in runSettings file, it is not able to log. Is there any workaround for this scenario. Any suggestions.