feat(config/travis): configurable regexes to parse artifacts #142

srvaroa · 2016-12-16T16:47:02Z

travis: configurable regexes to parse artifacts

Offer configurable regexes to find RPM/DEB artifacts pushed to
artifactory as the default ones only match on the output of the "art"
CLI, but wouldn't on that of any other tools. The result is a failed
Spinnaker pipeline when no artifact is found.

With this change users may add a list of arbitrary regexes that will be
used to parse build logs. An example is provided that corresponds to
Gradle maven-publish, which would match on the following line (taken
from a real build, with project names changed):

    Upload https://artifacts.my.org/artifactory/yum-local/some/project/package-0.1.0.20161216.g1b1c767.rpm

srvaroa · 2016-12-16T17:16:01Z

Sorry for the multiple pushes, I am unable to build locally due to:

What went wrong:
A problem occurred evaluating root project 'igor'.

Failed to apply plugin [class 'com.netflix.spinnaker.gradle.project.SpinnakerProjectConventionsPlugin']
index is out of range 0..-1 (index = 0)

srvaroa · 2016-12-23T08:26:22Z

@tomaslin @gardleopard let me know if this needs any improvements, thanks!

gardleopard · 2017-01-09T12:53:00Z

I would like it to keep the one there as the default and have a configuration override for adding custom parsers. ie some configurable parser setting for each travis server in https://github.com/spinnaker/igor/blob/master/igor-web/config/igor.yml

srvaroa · 2017-01-09T12:54:35Z

Thanks @gardleopard, that makes sense. I have this in my queue and will refactor as you suggest as soon as possible.

srvaroa · 2017-03-04T16:52:30Z

@gardleopard @tomaslin let me know how this looks now

gardleopard

This looks like a good solution to me, nice work @srvaroa !

I would like you to add an example of it into this file as well: https://github.com/spinnaker/igor/blob/master/igor-web/config/igor.yml#L17

gardleopard · 2017-03-04T17:57:44Z

...web/src/main/groovy/com/netflix/spinnaker/igor/travis/client/logparser/ArtifactParser.groovy

+    static List<GenericArtifact> getArtifactsFromLog(String buildLog, Iterable<String> additionalRegexes) {
+        final List<GenericArtifact> artifacts = new ArrayList<GenericArtifact>()
+        final Set<String> regexes = new HashSet<>()
+        regexes.addAll(DEFAULT_REGEXES)


I would prefer if injected regexes replaces the default ones. In some cases you would want to not use the standard regexes.

gardleopard · 2017-03-04T18:00:09Z

README.md

-Igor requires redis server to be up and running.
-
-Start Igor via `./gradlew bootRun`. Or by following the instructions using the [Spinnaker installation scripts](https://www.github.com/spinnaker/spinnaker).
+# Igor


Why is all this text replaced? You have not changed it all.

I didn't realize the file has DOS line endings and I have ff=UNIX in my vimrc. I'll respect the current settings.

gardleopard · 2017-03-04T18:00:52Z

README.md

+by using a `regexes` list:
+
+```
+# regexes:


I would move this up to travis so the placement for this is clear.

srvaroa · 2017-03-05T19:21:08Z

@gardleopard I had the example already at igor-web/config/igor.yml but let me know if I'm missing the right place.

tomaslin · 2017-03-21T04:21:58Z

is there any updates on this issue @gardleopard ?

gardleopard · 2017-04-20T12:10:57Z

Needs tests for parsing the yaml configuration, could not get it to work in our environment.

I tried with:

  regexes:
    - /Uploading artifact: https?:\/\/.+\/(.+\.(deb|rpm)).*/
    - /Successfully pushed (.+\.(deb|rpm)) to .*/
    - /Uploading artifact:\s+.*?([^\/]+\.(deb|rpm)).*/

srvaroa · 2017-04-20T13:54:08Z

Thanks @gardleopard, I will add some extra checks.

srvaroa · 2017-04-23T18:12:26Z

@gardleopard I have been starting the service with gradle run and different values in igor.yml. I think the mistake was just in the sample values provided, Groovy accepts / when declaring string literals so they are fine in the test. But in the yaml they are taken as part of the string. Just removing the surrounding / gets the regexes correctly loaded into the TravisService.

I updated this in igor.yml and made minor changes to the PR, but it's mostly as-is. Could you try with any of these config values to verify?

- Uploading artifact: https?:\/\/.+\/(.+\.(deb|rpm)).*
- Successfully pushed (.+\.(deb|rpm)) to .*
- Uploading artifact:\s+.*?([^\/]+\.(deb|rpm)).*

(Unrelated to this PR, but I noticed that the sample values for decoratorRegex and identifierRegex in igor.yml may have the same problem.)

gardleopard · 2017-04-24T12:12:49Z

This is good to go for me.
The examples above do not actually work it needs to be

    - Successfully pushed (.+\.(deb|rpm)) to .*
    - 'Uploading artifact:\s+.*?([^\/]+\.(deb|rpm)).*'

The : is interpreted by yaml and has to be quoted. I think as long as you are quoting the regex this will just work.

@tomaslin This is PR is ready to be merged in my opinion.

tomaslin · 2017-04-25T20:22:20Z

LGTM

tomaslin · 2017-04-25T20:23:25Z

Can you change the commit message so it fits within the spinnaker commit format? http://www.spinnaker.io/docs/how-to-submit-a-patch#section-commit-message-conventions

Offer configurable regexes to find RPM/DEB artifacts pushed to artifactory as the default ones only match on the output of the "art" CLI, but wouldn't on that of any other tools. The result is a failed Spinnaker pipeline when no artifact is found. With this change users may add a list of arbitrary regexes that will be used to parse build logs. An example is provided that corresponds to Gradle maven-publish, which would match on the following line (taken from a real build, with project names changed): Upload https://artifacts.my.org/artifactory/yum-local/some/project/package-0.1.0.20161216.g1b1c767.rpm

srvaroa · 2017-04-25T22:58:29Z

Thanks @tomaslin , @gardleopard - I've updated the PR & commit, let me know if I missed any detail.

srvaroa force-pushed the match-gradle-output branch 3 times, most recently from 0a00091 to 2ed0d79 Compare December 16, 2016 17:13

srvaroa force-pushed the match-gradle-output branch from 2ed0d79 to c4cef9d Compare December 23, 2016 08:23

srvaroa force-pushed the match-gradle-output branch from c4cef9d to e1e5b90 Compare March 4, 2017 16:51

srvaroa changed the title ~~Detect artifacts pushed with gradle plugin~~ travis: configurable regexes to parse artifacts Mar 4, 2017

gardleopard reviewed Mar 4, 2017

View reviewed changes

srvaroa force-pushed the match-gradle-output branch from e1e5b90 to 8185869 Compare March 5, 2017 19:19

srvaroa force-pushed the match-gradle-output branch from 8185869 to 1842e5e Compare March 31, 2017 08:25

srvaroa force-pushed the match-gradle-output branch from 1842e5e to f8abc51 Compare April 23, 2017 17:02

srvaroa force-pushed the match-gradle-output branch from f8abc51 to b025018 Compare April 25, 2017 22:57

srvaroa changed the title ~~travis: configurable regexes to parse artifacts~~ feat(config/travis): configurable regexes to parse artifacts Apr 25, 2017

tomaslin merged commit 25fe32f into spinnaker:master Apr 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(config/travis): configurable regexes to parse artifacts #142

feat(config/travis): configurable regexes to parse artifacts #142

srvaroa commented Dec 16, 2016 •

edited

Loading

srvaroa commented Dec 16, 2016

srvaroa commented Dec 23, 2016

gardleopard commented Jan 9, 2017

srvaroa commented Jan 9, 2017

srvaroa commented Mar 4, 2017

gardleopard left a comment

gardleopard Mar 4, 2017

gardleopard Mar 4, 2017

srvaroa Mar 5, 2017

gardleopard Mar 4, 2017

srvaroa commented Mar 5, 2017

tomaslin commented Mar 21, 2017

gardleopard commented Apr 20, 2017 •

edited

Loading

srvaroa commented Apr 20, 2017

srvaroa commented Apr 23, 2017

gardleopard commented Apr 24, 2017 •

edited

Loading

tomaslin commented Apr 25, 2017

tomaslin commented Apr 25, 2017

srvaroa commented Apr 25, 2017

feat(config/travis): configurable regexes to parse artifacts #142

feat(config/travis): configurable regexes to parse artifacts #142

Conversation

srvaroa commented Dec 16, 2016 • edited Loading

srvaroa commented Dec 16, 2016

srvaroa commented Dec 23, 2016

gardleopard commented Jan 9, 2017

srvaroa commented Jan 9, 2017

srvaroa commented Mar 4, 2017

gardleopard left a comment

Choose a reason for hiding this comment

gardleopard Mar 4, 2017

Choose a reason for hiding this comment

gardleopard Mar 4, 2017

Choose a reason for hiding this comment

srvaroa Mar 5, 2017

Choose a reason for hiding this comment

gardleopard Mar 4, 2017

Choose a reason for hiding this comment

srvaroa commented Mar 5, 2017

tomaslin commented Mar 21, 2017

gardleopard commented Apr 20, 2017 • edited Loading

srvaroa commented Apr 20, 2017

srvaroa commented Apr 23, 2017

gardleopard commented Apr 24, 2017 • edited Loading

tomaslin commented Apr 25, 2017

tomaslin commented Apr 25, 2017

srvaroa commented Apr 25, 2017

srvaroa commented Dec 16, 2016 •

edited

Loading

gardleopard commented Apr 20, 2017 •

edited

Loading

gardleopard commented Apr 24, 2017 •

edited

Loading