Pipeline parameters misinterpretation #11707

Deunze · 2019-01-23T11:16:26Z

Hello,
I'm faced with an issue related to pipeline parameters and artifacts parameters:
I have two pipelines. The first one is executed after a new revision of code (pulling). The second one use artifacts as build materials of the first pipeline.
In the first I have to collect artifacts and for that I use Build artifact parameter on a job:
Source: #{WORKING_DIRECTORY}/{Runtime}/Release/*.dll
Destination: Build/{Runtime}/Release
In the second pipeline directories are not created correctly. In fact I have:
Build/{Runtime/Release}
I was try several techniques to try to escape special chraracters ({}) but not succeed to resolve this.
The #{WORKING_DIRECTORY} is filled with a folder name. I don't see anything relevant about it in the documentation.
So my question is why the path is transformed in {Runtime/Release}? Is it a bug in parameters parsing of GoCD or we have to escape folders like {Runtime} in the other manner?

Deunze · 2019-01-23T12:03:28Z

Oh sorry I forgot to describe my environment:

Server Go Version: 18.7.0 (on Windows server 2012 R2)
Build agent (Windows 7)

arvindsv · 2019-01-23T15:24:25Z

@Deunze It's not clear what the question is. Do you mind showing some real snippets from the config and explaining what is happening?

Deunze · 2019-01-24T13:58:58Z

I have this configuration for the first pipeline:

<pipeline name="XXBuild" labeltemplate="${COUNT}_#{PROJECT_NAME}_V#{PROJECT_VERSION}.${P4_XXProj_2}_#{PROJECT_LABEL}" lockBehavior="unlockWhenFinished">
      <params>
        <param name="PROJECT_NAME">XXProj</param>
        <param name="PROJECT_VERSION">3.0.0</param>
        <param name="WORKING_DIRECTORY">XXProj</param>
        <param name="MS_BULD_VERSION">v4.0\V140</param>
        <param name="BUILD_PLATFORM">x64</param>
        <param name="PROJECT_LABEL">IncrementalBuild</param>
        <param name="BUILD_TARGET">build</param>
        <param name="CPU_COUNT">4</param>
        <param name="BUILD_CONFIG">Release</param>
      </params>
      <materials>
        <p4 port="localhost:1666" username="user" encryptedPassword="secret" useTickets="true" materialName="P4_XXProj_2">
          <view><![CDATA[//XStreams/XXProj/... //XStreams/XXProj/...]]></view>
        </p4>
      </materials>
      <stage name="Build" cleanWorkingDir="true">
        <jobs>
          <job name="MSBuild">
            <tasks>
              <exec command="%WINDIR%\Microsoft.NET\Framework64\v4.0.30319\MsBuild.exe" workingdir="#{WORKING_DIRECTORY}">
                <arg>/p:VCTargetsPath="%PROGRAMFILES(x86)%\MSBuild\Microsoft.Cpp\#{MS_BULD_VERSION}\\";Configuration="#{BUILD_CONFIG}";Platform="#{BUILD_PLATFORM}"</arg>
                <arg>/t:#{BUILD_TARGET}</arg>
                <arg>/m:#{CPU_COUNT}</arg>
                <arg>All_Projects.sln</arg>
                <runif status="passed" />
              </exec>
            </tasks>
            <artifacts>
              <artifact type="build" src="#{WORKING_DIRECTORY}/{Build}/*/#{BUILD_CONFIG}/*.dll" dest="Build" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/{Build}/*/#{BUILD_CONFIG}/*.exe" dest="Build" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/BuildTools" dest="Build" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/Testing" dest="Build" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/TestLib" dest="Build" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/{Runtime}/Release/*.dll" dest="Build/{Runtime}/Release" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/{Runtime}/Release/*.exe" dest="Build/{Runtime}/Release" />
              <artifact type="build" src="#{WORKING_DIRECTORY}/*/InputOutputData" dest="Build" />
            </artifacts>
          </job>
        </jobs>
      </stage>
    </pipeline>

So as described it must store artifacts files in "Build/{Runtime}/Release".
Here is the second pipeline:

<pipeline name="XXTest" labeltemplate="${COUNT}_#{PROJECT_NAME}_Vx_#{PROJECT_LABEL}" lockBehavior="unlockWhenFinished">
  <params>
    <param name="PROJECT_NAME">XX</param>
    <param name="PROJECT_VERSION">3.0.0</param>
    <param name="WORKING_DIRECTORY">Build</param>
    <param name="GTEST_FILTER">--gtest_filter=-*.*_nightly:*.*_OpenGL</param>
    <param name="PROJECT_LABEL">IncrementalTest</param>
  </params>
  <materials>
    <pipeline pipelineName="XXBuild" stageName="Build" materialName="XXBuild" />
  </materials>
  <stage name="Test" fetchMaterials="false">
    <jobs>
      <job name="UnitTest">
        <tasks>
          <fetchartifact artifactOrigin="gocd" srcdir="Build" pipeline="XXBuild" stage="Build" job="MSBuild">
            <runif status="passed" />
          </fetchartifact>
          <exec command="BuildTools\GoCD\RunUnitTests.bat" workingdir="#{WORKING_DIRECTORY}">
            <arg>"#{GTEST_FILTER}"</arg>
            <runif status="passed" />
          </exec>
        </tasks>
        <artifacts>
          <artifact type="test" src="#{WORKING_DIRECTORY}/TestReports/**" dest="testoutput" />
        </artifacts>
      </job>
    </jobs>
  </stage>
</pipeline>

But when build materials are stored in the working directory of the pipeline I have a bad sub directory construction:
Build/{Runtime/Release}/...
Instead of Build/{Runtime}/Release/...

ankitsri11 · 2019-01-25T00:37:44Z

I was able to replicate the behavior with the below configuration:

  <pipeline name="Sample">
     <materials>
       <git url="https://my.github.repo/" />
     </materials>
     <stage name="defaultStage">
       <jobs>
         <job name="defaultJob">
           <tasks>
             <exec command="dir" />
           </tasks>
           <artifacts>
             <artifact type="build" src="test.xml" dest="Build/{Runtime}/Release" />
           </artifacts>
         </job>
       </jobs>
     </stage>
   </pipeline>

After running the pipeline, I could see below folder created under artifact directory:

C:\Program Files (x86)\Go Server\artifacts\pipelines\Sample\2\defaultStage\1\defaultJob\Build\{Runtime\Release}

Here are the logs from go-agent logs for the same job:

2019-01-24 18:08:05,131 INFO  [scheduler-2] DefaultGoPublisher:95 - Agent [windows-server-2012-r2, 127.0.0.1, 58a5b411-39d7-4971-93cf-d51fdc0b4a69] is reporting status [Preparing] to Go Server for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:08:06,834 INFO  [scheduler-2] HttpService:138 - Got back 200 from server
2019-01-24 18:08:06,834 INFO  [scheduler-2] DefaultGoPublisher:95 - Agent [windows-server-2012-r2, 127.0.0.1, 58a5b411-39d7-4971-93cf-d51fdc0b4a69] is reporting status [Building] to Go Server for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:08:06,834 INFO  [scheduler-2] CommandLine:366 - Running command: cmd /c dir
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 -  Volume in drive C is Windows 2012 R2
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 -  Volume Serial Number is A099-33E6
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 -  Directory of C:\Program Files (x86)\Go Agent\pipelines\Sample
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM    <DIR>          .
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM    <DIR>          ..
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM                42 .travis.yml
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM             2,345 mingle_outlook.rb
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM               358 README.md
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM            91,532 test.xml
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 - 01/24/2019  06:00 PM               802 update_param.sh
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 -                5 File(s)         95,079 bytes
2019-01-24 18:08:06,850 INFO  [Thread-48] CommandLine:51 -                2 Dir(s)  41,371,242,496 bytes free
2019-01-24 18:08:06,975 INFO  [scheduler-2] HttpService:138 - Got back 200 from server
2019-01-24 18:08:06,975 INFO  [scheduler-2] DefaultGoPublisher:101 - Agent [windows-server-2012-r2, 127.0.0.1, 58a5b411-39d7-4971-93cf-d51fdc0b4a69] is reporting build result [Passed] to Go Server for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:08:07,006 INFO  [scheduler-2] HttpService:138 - Got back 200 from server
2019-01-24 18:08:07,006 INFO  [scheduler-2] DefaultGoPublisher:95 - Agent [windows-server-2012-r2, 127.0.0.1, 58a5b411-39d7-4971-93cf-d51fdc0b4a69] is reporting status [Completing] to Go Server for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:08:07,006 INFO  [scheduler-2] ArtifactsPublisher:69 - Pluggable metadata folder is empty.
2019-01-24 18:08:07,052 INFO  [scheduler-2] HttpService:71 - Uploading file [C:\Program Files (x86)\Go Agent\data\cruise-76218dd7-970f-4a35-8832-8042fd7d3c3f\d6f028ff-a312-457d-949a-0d94f050e78f\test.xml.zip] to url [https://127.0.0.1:8154/go/remoting/files/Sample/2/defaultStage/1/defaultJob/Build/%7BRuntime%7D/Release?attempt=1&buildId=2]
2019-01-24 18:08:07,115 INFO  [scheduler-2] HttpService:138 - Got back 201 from server
2019-01-24 18:08:07,115 INFO  [scheduler-2] DefaultGoPublisher:107 - Agent [windows-server-2012-r2, 127.0.0.1, 58a5b411-39d7-4971-93cf-d51fdc0b4a69] is reporting build result [Passed] to Go Server for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:08:07,147 INFO  [scheduler-2] HttpService:138 - Got back 200 from server
2019-01-24 18:08:07,193 INFO  [scheduler-2] DefaultGoPublisher:89 - Stopping Transmission for Build [Sample/2/defaultStage/1/defaultJob/2]
2019-01-24 18:12:31,207 INFO  [scheduler-2] HttpService:138 - Got back 200 from server

Job console log output:

##|18:08:06.819 [go] Start to build Sample/2/defaultStage/1/defaultJob on windows-server-2012-r2 [C:\Program Files (x86)\Go Agent]
!!|18:08:06.834 [go] Task: dir
&1|18:08:06.850  Volume in drive C is Windows 2012 R2
&1|18:08:06.850  Volume Serial Number is A099-33E6
&1|18:08:06.850 
&1|18:08:06.850  Directory of C:\Program Files (x86)\Go Agent\pipelines\Sample
&1|18:08:06.850 
&1|18:08:06.850 01/24/2019  06:00 PM    <DIR>          .
&1|18:08:06.850 01/24/2019  06:00 PM    <DIR>          ..
&1|18:08:06.850 01/24/2019  06:00 PM                42 .travis.yml
&1|18:08:06.850 01/24/2019  06:00 PM             2,345 mingle_outlook.rb
&1|18:08:06.850 01/24/2019  06:00 PM               358 README.md
&1|18:08:06.850 01/24/2019  06:00 PM            91,532 test.xml
&1|18:08:06.850 01/24/2019  06:00 PM               802 update_param.sh
&1|18:08:06.850                5 File(s)         95,079 bytes
&1|18:08:06.850                2 Dir(s)  41,371,242,496 bytes free
?0|18:08:06.959 [go] Task status: passed (125 ms)
j0|18:08:06.990 [go] Current job status: passed
##|18:08:07.006 [go] Start to create properties Sample/2/defaultStage/1/defaultJob on windows-server-2012-r2 [C:\Program Files (x86)\Go Agent]
ar|18:08:07.006 [go] Start to upload Sample/2/defaultStage/1/defaultJob on windows-server-2012-r2 [C:\Program Files (x86)\Go Agent]
ar|18:08:07.038 [go] Uploading artifacts from C:\Program Files (x86)\Go Agent\pipelines\Sample\test.xml to Build/{Runtime}/Release
ex|18:08:07.115 [go] Job completed Sample/2/defaultStage/1/defaultJob on windows-server-2012-r2 [C:\Program Files (x86)\Go Agent]

ankitsri11 · 2019-01-25T22:47:51Z

I was able to replicate this behaviour across all OS.

Later, when I tried uploading artifact using the create artifact API (https://api.gocd.org/current/#create-artifact) was unable to replicate the issue.

Pipeline Configuration:

<pipeline name="Sample">
      <materials>
        <git url="https://my.github.repo" />
      </materials>
      <stage name="defaultStage">
        <jobs>
          <job name="defaultJob">
            <tasks>
              <exec command="ls" />
            </tasks>
            <artifacts>
              <artifact type="build" src="test.xml" dest="build/{runtime}/release" />
            </artifacts>
          </job>
        </jobs>
      </stage>
    </pipeline>

API call:

curl 'http://localhost:8153/go/files/Sample/1/defaultStage/1/defaultJob/build/{runtime}/release' \
      -H 'Confirm: true' \
      -X POST \
      -F 'zipfile=@some_file.zip'

However, when I appended the path using %7B ... %7D instead of {}, the issue was replicable.

API call:

curl 'http://localhost:8153/go/files/Sample/1/defaultStage/1/defaultJob/build1/%7Bruntime%7D/release/abcd' \
      -H 'Confirm: true' \
      -X POST \
      -F 'zipfile=@some_file.zip'

arvindsv · 2019-01-29T23:37:29Z

This is a bug in urlrewrite, similar to this one.

This call causes this problem when passed in a string with braces in it.

FunctionReplacer.replace("escape:{runtime}/release/abcd") => "{runtime/release/abcd}"

... because it has a regex which is not greedy.

The matching filter in GoCD code is:

gocd/server/webapp/WEB-INF/urlrewrite.xml

Line 602 in 0ba4950

    
           <to type="forward">/repository/restful/artifact/%{attribute:_method}/%{attribute:_type}?pipelineName=$1&amp;pipelineCounter=$2&amp;stageName=$3&amp;stageCounter=$4&amp;buildName=$5&amp;filePath=${escape:$6}</to>

Replacing ${escape:$6} with just $6 fixes it, but it might have other side-effects.

@ketan might know more.

arvindsv · 2019-01-29T23:38:08Z

@Deunze Easiest thing currently would be to remove the braces for now. Even by changing/escaping that path, it looks like it'll not get the right path.

ketan · 2019-01-30T02:01:05Z

Replacing ${escape:$6} with just $6 fixes it, but it might have other side-effects.

@ketan might know more.

It does fix, but introduces other problems, don't recall what those were.

Deunze · 2019-02-01T08:31:33Z

Thanks guys,
Hope it will fixed in a future GoCD release.

chadlwilson mentioned this issue Aug 11, 2021

Bump urlrewritefilter from 3.2.0 to 4.0.4 #9357

Draft

chadlwilson transferred this issue from gocd/docs.go.cd Jun 22, 2023

chadlwilson added bug apis artifacts labels Jun 22, 2023

chadlwilson mentioned this issue Jan 26, 2024

Can't collect artifacts from a folder containing a "%" or ";" in its name #12466

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipeline parameters misinterpretation #11707

Pipeline parameters misinterpretation #11707

Deunze commented Jan 23, 2019

Deunze commented Jan 23, 2019

arvindsv commented Jan 23, 2019

Deunze commented Jan 24, 2019 •

edited by arvindsv

ankitsri11 commented Jan 25, 2019 •

edited

ankitsri11 commented Jan 25, 2019 •

edited

arvindsv commented Jan 29, 2019

arvindsv commented Jan 29, 2019

ketan commented Jan 30, 2019

Deunze commented Feb 1, 2019

Pipeline parameters misinterpretation #11707

Pipeline parameters misinterpretation #11707

Comments

Deunze commented Jan 23, 2019

Deunze commented Jan 23, 2019

arvindsv commented Jan 23, 2019

Deunze commented Jan 24, 2019 • edited by arvindsv

ankitsri11 commented Jan 25, 2019 • edited

ankitsri11 commented Jan 25, 2019 • edited

arvindsv commented Jan 29, 2019

arvindsv commented Jan 29, 2019

ketan commented Jan 30, 2019

Deunze commented Feb 1, 2019

Deunze commented Jan 24, 2019 •

edited by arvindsv

ankitsri11 commented Jan 25, 2019 •

edited

ankitsri11 commented Jan 25, 2019 •

edited