Question: how to extract archive contents during dependency resolution

[chefhoobajoob]

We are using the latest version of the teamcity plugin for Artifactory and TC version 9.1.6.

I have a c++ project with an Artifactory dependency that consists of a zip file containing the header/lib file system of the dependency.

I can find no documentation anywhere on how to use the TeamCity Artifactory plugin to extract the entire contents of the archive into a directory of my choosing.

I have tried the syntax below using the plugin's Custom build dependencies field, but the plugin complains that the ** wildcard is illegal. How can I accomplish this operation during dependency resolution prior to the build?

*:path/to/dependency-01.zip!** => ../Dependency-01

[eyalbe4]

Hi @chefhoobajoob, the TeamCity Artifactory Plugin does not currently include the option to extract zip files after they are downloaded from Artifactory. You can probably unzip the file locally using a shell script that is executed after the download. If your download is more complicated and requires more flexibility, you can consider embedding JFrog CLI into a script that handles your Artifactory download or upload.

[chefhoobajoob]

Dang. Ok, so since I'm new to both TC and Artifactory, would this be done by inserting a build step ahead of the step that performs the compilation of the dependent component (and afterwards to create zip archives and publish)?

Something like this?

• build step 1: a command line or shell builder step configured only to resolve artifact dependencies and retrieve the archive into some directory, and the body of which unzips into the expected location
• build step 2: a visual studio builder that does not resolve or publish Artifactory dependencies, but compiles the dependent now that the dependencies have been extracted to the expected location
• build step 3: a command line or shell builder step configured only to publish a zip archive to artifactory whose body just creates the zip archive from the output of the previous build step

[eyalbe4]

@chefhoobajoob, Yes, this is something you can implement and will work fine. If your CI flow permits that, please also consider not to store a zip file in Artifactory, but have all of your build files deployed unzipped, by using the "Custom published artifacts" fields. That allows you to later perform operations on all artifacts of a specific build (such as Artifactory Promotion). You can also publish Build Info to Artifactory for every build and have Artifactory manage this information for you.

[chefhoobajoob]

@eyalbe4: thank you for your feedback. This issue can probably be closed, but I do have a couple questions about publishing to Artifactory. So I updated my build as you suggested and the first build worked just fine, but all subsequent publishing attempts fail with the same error:

[08:53:38][Step 2/2] Failed while reading the response from: PUT http://artifactory.service.net/libs-snapshot-local/Component/Source/Component.h;vcs.revision=bfcb1a5bf550ec91f3e6c1060d78ff0e67208d31;build.timestamp=1461588802506;build.name=Some%20::%20Core%20::%20component%20::%201.1.0;build.number=1.1.0.11 HTTP/1.1
[08:53:38]
[Step 2/2] org.codehaus.jackson.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: org.apache.http.conn.EofSensorInputStream@33f232; line: 1, column: 2]
    at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)

...I can't find any way to publish the results of a second build. It seems to be thinking that I'm trying to replace or update results from the first build. Strangely, I see similar errors in the first build, but for some reason it succeeded, and I can see the published results in Artifactory. Not so with any later build.

What could be causing this error?

This is an older version of Artifactory: 2.6.6. We are in the process of upgrading - could this be a problem related to some kind of incompatibility between the plugin and the server?

[eyalbe4]

@chefhoobajoob, it looks like Artifactory, or more likely a proxy you have before your Artifactory instance is returning an HTML response, and the client fails because it expects a json response. I'd try to see if there's anything in the Artifactory logs and if the failing request is not included in Artifactory's request log, check the proxy logs. For troubleshooting, you can also try and have your TeamCity agent bypass the proxy and send the requests to the URL Artifactory directly.

[chefhoobajoob]

@eyalbe4: I don't think there's a proxy between Artifactory and the TC agents, but I will check

[chefhoobajoob]

@eyalbe4 : I can confirm that there's no proxy, but I can't get at the Artifactory logs yet because I'm not the admin, so I hope to learn more later.

What I'm noticing though, is that Artifactory seems to require me to manually include version and build information in the path to published artifacts, or else it thinks I'm trying to overwrite existing artifacts.

That is, in my component build, rather than defining the published artifacts like so:

Include/**/* => MyComponent/Include
Source/**/*.h => MyComponent/Source
Lib/**/*.lib => MyComponent/Lib

...I have to define the paths like this:

Include/**/* => MyOrg/MyGroup/MyComponent/1.1.0/%build.number%/MyComponent/Include
Source/**/*.h => MyOrg/MyGroup/MyComponent/1.1.0/%build.number%/MyComponent/Source
Lib/**/*.lib => MyOrg/MyGroup/MyComponent/1.1.0/%build.number%/MyComponent/Lib

...which is not only more verbose, but I have to change it for different build configurations, so it's error-prone too.

Am I missing something? Is there a way to avoid this, or for the plugin to ensure that artifacts from different builds of the same component don't step on each other without requiring this manual configuration?

[ElanHasson]

Would be great if the plugin supported extraction.

[eyalbe4]

@ElanHasson,
We're planning to add extraction (explode) of archive files when uploading and downloading from Artifactory using File Specs. We will extend the File Specs schema to include this option.

[RichardGe]

Hi, is this still a missing feature ?

[eyalbe4]

@RichardGe,
This can be achieved by using JFrog CLI's jf rt download command. The command includes the --explode option to expload an archive after it is downloaded. We recommend using JFrog CLI within your TeamCity builds to achieve this.