issue to copy all test data files to output folder using runsettings file, as supported by testsettings file

[anshuls01]

Description

We are following Microsoft DevOps Blog Link to migrate from MsTest to MsTestV2
Earlier Unit Test Scripts using the testsettings files, testsettings deployment setting looks like as below:
<Deployment> <DeploymentItem filename="WebUITestProject\Data\GE1\US\" /> <DeploymentItem filename="Data\GE1\US\" /> </Deployment>
With the above setting, MsTest framework copy all the test data files from “US” folder to output folder.
And the Datasource attribute of test case looks like:
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "testdatafile.xml", "node", DataAccessMethod.Sequential)]

And the testdata is accessible through TestContext DataRow property.

Test project contains several small Testdata files for different regions and test environment, and to switch between the different regions and test environment, only change is to select different testsettings file.
e.g. Testdata files maintain like the below structure, where GE1, GE2 and GE3 indicates the Test Environment and UK & US are indication for regions

To migrate the testsettings file I followed the Visual studio SettingsMigrator blog, and this link talks about to configure the unit test by using runsettings file
After migration to MsTestV2 and with runsettings file, Test data files for a designated folder doesn’t copy to output folder,
even if include multiple Deployment tag(one for each file) “/RunSettings/LegacySettings/Deployment”, files doesn’t copy to output folder.
I tried to give the full path as per this link or Relative path but the above didn’t work.

For the time being, Test scripts has DeploymentItem and DataSource attributes
[DeploymentItem(@"Data\GE1\US\PartnerPortal_US.xml", "TestData")] [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\TestData\\PartnerPortal_US.xml", "PartnerPortal", DataAccessMethod.Sequential)]

The problem with the above approach is, all DeploymentItem path require change to run the automation for different test environment or region. That further includes the multiple builds.

Steps to reproduce

Unzip the attach solution
WebUITestProject.zip
Put a break point on the first line of testcase, Debug the test case. Once hit the breakpoint
Check output folder, output folder contains only the files mentioned at DeploymentItem attribute, no file from "WebUITestProject\Data\GE1\US" to output folder.

Expected behavior

Test Data files should copy to the output folder as specified in runsettings

Actual behavior

Test Data files don't copy, only deploymentItem attribute is working.

Environment

Vs 2017
MsTest V2
Windows 10

[karanjitsingh]

@anshuls01, you need to set the runsettings file in VS IDE to be able to run this,

If you're running this through vstest.console.exe from developer command prompt then

.\Common7\IDE\Extensions\TestPlatform\vstest.console.exe --ResultsDirectory:\path\to\results_directory --Settings:\path\to\US_GE1_TestData.runsettings  \path\to\WebUITestProject.dll

will work too

[anshuls01]

Hello Karanjit,
I tried using that only, it's not working.
Testcase output folder contains all the required dll, xml file mentioned in DeploymentItem attribute,
only the files I am referring through runsettings are missing.
<LegacySettings> <Deployment> <DeploymentItem filename="WebUITestProject\Data\GE1\US\"/> </Deployment> </LegacySettings>

just for the example, I kept only one XML file under US folder, there could be multiple and should copy to the output folder.

let say DataSource attribute looks like:
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\PartnerPortal_US.xml", "PartnerPortal", DataAccessMethod.Sequential)]

as I know here |DataDirectory| is the testcase output folder.
If all the files get copied to output folder, I can run the same test for multiple regions and TestEnvironment without creating new build or change any code. The only change required to select the different runsettings file.

[karanjitsingh]

@anshuls01, just to clarify the output folder you're talking about. Do you mean the Deploy/out folder that is created in the test results directory or the Bin/Debug directory?

[anshuls01]

@karanjitsingh Deploy/out folder that is created in the test results directory

[karanjitsingh]

@anshuls01, I think I found the problem here

Try to remove <ForcedLegacyMode>True</ForcedLegacyMode> from your run settings, this made it work on my end

WebUITestProject.zip

[anshuls01]

@karanjitsingh thanks for the update, I tried the solution you have suggested but didn't work for me.

just elaborate little more:
Deploymentitem attribute of test case looks like:
[DeploymentItem(@"Data\GE1\US\PartnerPortal_US.xml", "TestData")]
and the file PartnerPortal_US.xml gets copied to out folder TestResults\Deploy_<userName> 2019-02-01 13_30_29\Out\TestData folder

the solution you have shared have total 3 files under Data\GE1\US, with the runsettings deployment item no files copied to ``TestResults\Deploy_ 2019-02-01 13_30_29\Out` folder.

I tried different path like:
<DeploymentItem filename=".\WebUITestProject\Data\GE1\US\PartnerPortal_US_2.xml" /> <DeploymentItem filename=".\Data\GE1\US\PartnerPortal_US_2.xml" /> <DeploymentItem filename=".\Data\GE1\US\" /> <DeploymentItem filename=".\WebUITestProject\Data\GE1\US\" /> <DeploymentItem filename="WebUITestProject\Data\GE1\US\" /> <DeploymentItem filename="Data\GE1\US\" />

attaching the runsettings file and one of ``TestResults\Deploy_ 2019-02-01 13_30_29\Out` folder for your reference.

US_GE1_TestData.zip

I think we are close to the solution, required your little help to solve the issue.
In Microsoft blog i read, they were using the file full path, not sure here how I'll pass the relative path.

another work around could be, create few batch file to perform xCopy and include as setupscript in runsettings. there also we need a relative path. I'll try that solution and let you know the result.

Meanwhile please share your point of view on the above.

Thanks
Anshul

[karanjitsingh]

@anshuls01, this seems to be really weird behavior with consistent repro so i'll be marking this as a bug.

Just to clarify, the <LegacySettings> does not work with MSTestV2. I would suggest using the [DeploymentItem()] attribute only.

[anshuls01]

@karanjitsingh I tried with Batch script as well, but doesn't seems to be trigger at all.
If <LegacySettings> does not work with MsTestV2, it'll be a real challenge to migrate to V2.

[DeploymentItem()] alone doesn't solve the problem entirely and requires lots of change to run the same test script for multiple test environments (Where each test environments uses separate set of test data)

Is it possible to share the time line by when solution can be expected?

[karanjitsingh]

@anshuls01 as a workaround, you can use the following runsettings.

NOTE: This will cause all of the files in the bin directory to be copied to the out folder
This should help you with your scenario.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
  </RunConfiguration>
  <WebTestRunConfiguration testTypeId="4e7599fa-5ecb-43e9-a887-cd63cf72d207">
    <Browser name="Internet Explorer 9.0" MaxConnections="6">
      <Headers>
        <Header name="User-Agent" value="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" />
        <Header name="Accept" value="*/*" />
        <Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
        <Header name="Accept-Encoding" value="GZIP" />
      </Headers>
    </Browser>
  </WebTestRunConfiguration>
  <MSTest>
    <DeployTestSourceDependencies>false</DeployTestSourceDependencies>
    <DeploymentEnabled>True</DeploymentEnabled>
  </MSTest>
</RunSettings>

[anshuls01]

@karanjitsingh Yes it's working as you mentioned, still my original problem not solved by this.
the structure I got in output folder is like the below image:

even now if the same automation script needs to execute on different TestEnvironment, then [DataSource()] attribute value requires modification.

Is it possible to run a batch script (before testcase execution start) to copy the files from inner directory to output directory root? just thinking another workaround. still not sure how the parameter will pass to the batch script and how the batchscript get to know the output folder.

[karanjitsingh]

@anshuls01 you can get the results run directory through TestContext

[anshuls01]

@karanjitsingh Yes, that's an option, still I am away to solve the original problem.

Another possible solution I tried to Extend the test case execution at Test method level

only one problem I am facing here, TestContext is not accessible inside
public override TestResult[] Execute(ITestMethod testMethod)

if test context available here, i can configure the run settings to pass the parameter and do the deployment of necessary files to out directory before test case start.

[karanjitsingh]

@anshuls01, if you want to access TestContext inside a test method store it in a static variable during class initialize and access that variable in the method.

Edit: Apologies for accidentally closing

[anshuls01]

@karanjitsingh I am trying with TestMethod Extensions, till now it progressing good 👍
I am able to access the TestContext as you suggested 💯, earlier the same thing I was trying with public class, according to me it should work but didn't.

I'll keep you posted on progress

[anshuls01]

@karanjitsingh I created one TestMethod Extension (let say DeploymentExtension)

DeploymentExtension: copies the required test data files to Out directory, Test data files path mentioned in runsettings file.
<TestRunParameters> <Parameter name="DeploymentFolder" value="Data\GE1\US"/> </TestRunParameters>

the above extension is working fine, and able to copy the testdata files to Out folder

I run into a weired situation, on a TestMethod if specified [DeploymentExtension] attribute and [DataSource] attribute, [DataSource] attribute trigger first, due to that following error coming "The unit test adapter failed to connect to the data source or to read the data" and that error has to be, cause testdata files are not present in Out folder.

is there are any way to know which Attribute will evaluate first?

[anshuls01]

@karanjitsingh i think i got the Answer, if wrting the TestMethod attribute extension then tiggering the testcase ITestMethod.Invoke() will be the responsiblity of attribute extension, and that will be responsible to provide the test data to test case.
sorry for creating the confusion.

[anshuls01]

@karanjitsingh Hello Karan, I am able to resolve the issue for now by extended the Test Method and access the test context object. Please let me know once this issue fixed, it'll save a few lines of custom code force to add to handle the Test data file.

I've one more open issue to access the RunId property value, earlier in MsTest we use to get the run Id TestContext.Properties["__Tfs_TestRunId__"] and that run id is very crucial for us to track the execution through TFS services.

the above issue marked as an enhancement, still waiting to resolve this issue and blocker to move to MsTestV2.

Thanks
Anshul

[jayaranigarg]

@anshuls01 : In your repro project attached in the 'steps to reporoduce', the path of DeploymentItem is not relative to the runsettings file. Please try the path as

<DeploymentItem filename="WebUITestProject\Data\GE1\US\" />

Also, it seems you are trying to use MSTestV2. But please note that LegacySettings are not honoured by MSTestV2. So, I went ahead and moved the project to MSTestV1 and things seems to be working fine for me without need of suppling any [DeploymentItem] attribute over tests.

Attaching a working project here
WebUITestProject_Working.zip

Let me know if this works for you!

[ShreyasRmsft]

@anshuls01 the workarounds suggested are the only way in mstest v2. We will not be adding support for deployment item in runsettings/testsettings for mstest V2.

[prodoshkm]

@karanjitsingh Hello Karan, I am able to resolve the issue for now by extended the Test Method and access the test context object. Please let me know once this issue fixed, it'll save a few lines of custom code force to add to handle the Test data file.

I've one more open issue to access the RunId property value, earlier in MsTest we use to get the run Id TestContext.Properties["__Tfs_TestRunId__"] and that run id is very crucial for us to track the execution through TFS services.

the above issue marked as an enhancement, still waiting to resolve this issue and blocker to move to MsTestV2.

Thanks
Anshul

Hi Anshul,

I hope you have figured it out. If not, then try the following code. It works.

Add the following method in your Base class.

    [AssemblyInitialize()]
    public static void TestAssemblyInitialize(TestContext testContext)
    {
        string testDataSourceDirPath = testContext.Properties["TestDataDirectory"].ToString();
        DirectoryInfo sourceDir = new DirectoryInfo(testDataSourceDirPath);

        string testDataDestDirPath = ".\\TestData";
        DirectoryInfo destDir = new DirectoryInfo(testDataDestDirPath);

        foreach (FileInfo file in sourceDir.GetFiles("*.*", SearchOption.TopDirectoryOnly))
            file.CopyTo(Path.Combine(destDir.FullName, file.Name), true);
    }

And in your .runsettings file, add the following parameter:
<Parameter name="TestDataDirectory" value=".\Data\GE1\US" />

Hope this will solve your issue.