Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-38381] Pipeline remote build logging #18

Closed
wants to merge 53 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
664bd6b
Upgrade the plugin to the latest Parent POM and to 1.625.3 baseline.
oleg-nenashev Sep 16, 2016
3aa94a1
Fix incorrect Javadoc
oleg-nenashev Sep 16, 2016
68a2d1e
Restore injected tests and escape forms by default
oleg-nenashev Sep 16, 2016
8ccd669
Very first prototype of Remote logging from slaves
oleg-nenashev Sep 16, 2016
849c9cd
Workaround JENKINS-38291 - Do not report vars with dots to Elasticsearch
oleg-nenashev Sep 16, 2016
bbf3310
Attach unique Job IDs to the reports
oleg-nenashev Sep 16, 2016
679d53e
Hide secret in Logstash.
Sep 17, 2016
a7ee24a
Merge branch 'jw-hackaton-remote-logging' of https://github.com/oleg-…
Sep 17, 2016
73fed23
Save the progress.
oleg-nenashev Sep 17, 2016
969a7ae
Fix duplicated log messages.
Sep 17, 2016
8b23ae2
Pipeline support - Adapt BuildData and LogstashBuildWrapper
oleg-nenashev Sep 17, 2016
c0db1ca
KibanaEmbeddedLogAction: Fix symbols in the search query
oleg-nenashev Sep 17, 2016
7511154
Update the core to a Pipeline-compatible version
oleg-nenashev Sep 17, 2016
fe66fec
Semi-working Elastic-search based propagation
oleg-nenashev Sep 17, 2016
32db954
Fix the glitch with ArrayOutOfBounds in output
oleg-nenashev Sep 17, 2016
abbaadf
Remove debug output
oleg-nenashev Sep 17, 2016
2659d46
Merge branch 'pipeline-support' into jw-hackaton-remote-logging
oleg-nenashev Sep 17, 2016
2168eb0
Merge branch 'web-ui-log-viewer' into jw-hackaton-remote-logging
oleg-nenashev Sep 17, 2016
577d137
BuildData constructor should not fail with NPE when Jenkins instance …
oleg-nenashev Sep 17, 2016
b489e23
Ignore BuildData tests broken by Pipeline integration
oleg-nenashev Sep 17, 2016
f455aff
Add skeleton implementation for the External logging extension point
oleg-nenashev Sep 17, 2016
c301636
Basic support of Console log reporting for Pipeline
oleg-nenashev Sep 17, 2016
fe18d32
Fix printing of logs in Elasicsearch-based console
oleg-nenashev Sep 17, 2016
a61a7be
Elasticsearch-based console: Also display timestamp
oleg-nenashev Sep 17, 2016
8529515
Elasticsearch-based log: Report real data from the job (via query par…
oleg-nenashev Sep 17, 2016
8b75c99
Restore ConsoleAnnotator code
oleg-nenashev Sep 17, 2016
13bbfab
Add runtime build status balls to External Console Log actions.
oleg-nenashev Sep 18, 2016
2a5f92a
Allow posting logs at same second as the build start
oleg-nenashev Sep 18, 2016
0e502ed
Some fixes for progressive logs in Elasticsearch browser
oleg-nenashev Sep 18, 2016
4bcbe7b
use new LoggingMethod extension point.
Sep 19, 2016
2541097
Merge pull request #1 from xyan0607/jw-hackaton-remote-logging
oleg-nenashev Oct 4, 2016
7d92624
[JENKINS-38381] Prototype of using Logstash for a complete Pipeline b…
jglick Oct 14, 2016
bd5c9a2
Going back to 2.24 baseline. 2.19.2 would be fine when it comes out.
jglick Oct 14, 2016
fcb709f
jitpack, and linking to https://github.com/jenkinsci/maven-hpi-plugin…
jglick Dec 7, 2016
e886ea2
Demo sort of working.
jglick Dec 8, 2016
4a089e2
Work even in UTC.
jglick Dec 8, 2016
4d6b55e
Using millisecond granularity in messages, and sorting.
jglick Dec 8, 2016
9b8212f
Make sure source_host is sent.
jglick Dec 8, 2016
977bf05
TODO comment.
jglick Dec 8, 2016
7feaf75
Record the host sending each message, so it is clear which are logged…
jglick Dec 8, 2016
e168c03
Finally tracked down the missing-record bug.
jglick Dec 8, 2016
3f66386
Demonstrating password masking.
jglick Dec 8, 2016
b5fa447
Showing parallel work.
jglick Dec 8, 2016
9619531
hpi-plugin.version=1.121
jglick Dec 16, 2016
74e4379
Recording demo protocol.
jglick Dec 21, 2016
4f22bd3
getLog should take a long start argument.
jglick Dec 21, 2016
5c4d441
Refresh demo plugins.
jglick Dec 21, 2016
5888222
Giving up on jitpack for now.
jglick Jan 3, 2017
d00e7f5
Merge branch 'master' into logs-JENKINS-38381
jglick Feb 6, 2018
40008c9
Updated demo.
jglick Feb 6, 2018
5bb3486
Using updated dependencies, without merging _this_ with master.
jglick Jun 11, 2018
b75f573
Updated.
jglick Jun 11, 2018
3b11233
Updating Pipeline deps.
jglick Jun 11, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .mvn/extensions.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
<extensions xmlns="http://maven.apache.org/EXTENSIONS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/EXTENSIONS/1.0.0 http://maven.apache.org/xsd/core-extensions-1.0.0.xsd">
<extension>
<groupId>io.jenkins.tools.incrementals</groupId>
<artifactId>git-changelist-maven-extension</artifactId>
<version>1.0-beta-3</version>
</extension>
</extensions>
2 changes: 2 additions & 0 deletions .mvn/maven.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-Pconsume-incrementals
-Pmight-produce-incrementals
8 changes: 8 additions & 0 deletions demo/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
FROM jenkinsci/jenkins:2.121.1

ADD JENKINS_HOME /usr/share/jenkins/ref/

# https://github.com/docker/docker/issues/6119 workaround:
USER root
RUN chown -R jenkins.jenkins /usr/share/jenkins/ref
USER jenkins
1 change: 1 addition & 0 deletions demo/JENKINS_HOME/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
plugins
42 changes: 42 additions & 0 deletions demo/JENKINS_HOME/config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
<?xml version='1.0' encoding='UTF-8'?>
<hudson>
<disabledAdministrativeMonitors>
<string>jenkins.security.s2m.MasterKillSwitchWarning</string>
</disabledAdministrativeMonitors>
<version>1.0</version>
<numExecutors>1</numExecutors>
<mode>NORMAL</mode>
<useSecurity>true</useSecurity>
<authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
<securityRealm class="hudson.security.HudsonPrivateSecurityRealm">
<disableSignup>false</disableSignup>
<enableCaptcha>false</enableCaptcha>
</securityRealm>
<disableRememberMe>false</disableRememberMe>
<projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
<workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
<buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
<markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
<jdks/>
<viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
<myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
<clouds/>
<scmCheckoutRetryCount>0</scmCheckoutRetryCount>
<views>
<hudson.model.AllView>
<owner class="hudson" reference="../../.."/>
<name>All</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties class="hudson.model.View$PropertyList"/>
</hudson.model.AllView>
</views>
<primaryView>All</primaryView>
<slaveAgentPort>50000</slaveAgentPort>
<label></label>
<crumbIssuer class="hudson.security.csrf.DefaultCrumbIssuer">
<excludeClientIPFromCrumb>false</excludeClientIPFromCrumb>
</crumbIssuer>
<nodeProperties/>
<globalNodeProperties/>
</hudson>
18 changes: 18 additions & 0 deletions demo/JENKINS_HOME/credentials.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<?xml version='1.0' encoding='UTF-8'?>
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="credentials@2.1.10">
<domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
<entry>
<com.cloudbees.plugins.credentials.domains.Domain>
<specifications/>
</com.cloudbees.plugins.credentials.domains.Domain>
<java.util.concurrent.CopyOnWriteArrayList>
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl plugin="plain-credentials@1.3">
<scope>GLOBAL</scope>
<id>token</id>
<description></description>
<secret>Z9Ijvs1aPbFgmZ4SrVScn0fEdhM/KmrtvujXjtr1ij0=</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
</java.util.concurrent.CopyOnWriteArrayList>
</entry>
</domainCredentialsMap>
</com.cloudbees.plugins.credentials.SystemCredentialsProvider>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2.32
1 change: 1 addition & 0 deletions demo/JENKINS_HOME/jenkins.install.UpgradeWizard.state
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2.32
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
<?xml version='1.0' encoding='UTF-8'?>
<jenkins.model.JenkinsLocationConfiguration>
<adminAddress>address not configured yet &lt;nobody@nowhere&gt;</adminAddress>
<jenkinsUrl>http://localhost:8080/</jenkinsUrl>
</jenkins.model.JenkinsLocationConfiguration>
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<?xml version='1.0' encoding='UTF-8'?>
<jenkins.plugins.logstash.LogstashInstallation_-Descriptor plugin="logstash@1.2.1-SNAPSHOT">
<type>ELASTICSEARCH</type>
<host>http://elk</host>
<port>9200</port>
<username></username>
<password></password>
<key>/logstash/logs</key>
</jenkins.plugins.logstash.LogstashInstallation_-Descriptor>
27 changes: 27 additions & 0 deletions demo/JENKINS_HOME/jobs/sample/config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<?xml version='1.0' encoding='UTF-8'?>
<flow-definition plugin="workflow-job@2.10-SNAPSHOT">
<actions/>
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
<org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>
<triggers/>
</org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>
</properties>
<definition class="org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition" plugin="workflow-cps@2.23">
<script>parallel local: {
node(&apos;master&apos;) {
sh &apos;for x in 0 1 2 3 4 5 6 7 8 9; do echo $x; sleep 1; done&apos;
}
}, remote: {
node(&apos;remote&apos;) {
withCredentials([string(credentialsId: &apos;token&apos;, variable: &apos;TOKEN&apos;)]) {
sh &apos;echo receiving $TOKEN&apos;
sh &apos;for x in 0 1 2 3 4 5 6 7 8 9; do echo $x; sleep 1; done&apos;
}
}
}</script>
<sandbox>true</sandbox>
</definition>
<triggers/>
</flow-definition>
12 changes: 12 additions & 0 deletions demo/JENKINS_HOME/nodes/remote/config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<?xml version='1.0' encoding='UTF-8'?>
<slave>
<name>remote</name>
<description></description>
<remoteFS>/home/jenkins</remoteFS>
<numExecutors>1</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.slaves.JNLPLauncher"/>
<label></label>
<nodeProperties/>
</slave>
Binary file added demo/JENKINS_HOME/secrets/hudson.util.Secret
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions demo/JENKINS_HOME/secrets/master.key
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
6816dcc6d8d1cf2a87c53827f07fdf47f8775873dfe478a81922809ce22a9917670ea81dc7d5a20930b0998f54d347f3b1bd50ad6d61861b28fb02d50f728eb7b6e914b8ef5dd032feb02f0f618aba6f9d75fae4b797c946172f051b931a216b6abc0edbd9d8c75044f4b57b9477da33859f96902ee26051c77b8d50c03fd68c
9 changes: 9 additions & 0 deletions demo/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
plugins:
if [ \! -d ../target/test-classes/test-dependencies -o ../pom.xml -nt ../target/test-classes/test-dependencies/index ]; then mvn -f .. -DskipTests clean package; fi
rm -rf JENKINS_HOME/plugins
mkdir JENKINS_HOME/plugins
cp -v ../target/test-classes/test-dependencies/*.hpi ../target/*.hpi JENKINS_HOME/plugins

run: plugins
docker-compose rm -fv
docker-compose up --build --force-recreate jenkins elk
13 changes: 13 additions & 0 deletions demo/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
To run this demo:

1. `make run`
1. when that is started enough, `docker-compose up -d agent`
1. go to http://localhost:8080/job/sample/
1. **Build Now**
1. inspect **Console Output**
1. see console annotations in `Running on`__`Jenkins`__`in /var/jenkins_home/workspace/sample`
1. **hide**/**show** blocks
1. configure http://localhost:5601/ with `logstash*` and `@timestamp`
1. inspect **External log (Kibana)**
1. compare `sender` fields to `docker ps` container IDs
1. verify that passwords are masked
25 changes: 25 additions & 0 deletions demo/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
version: '2'
services:
elk:
image: sebp/elk:es240_l240_k460
ports:
- "127.0.0.1:5000:5000"
- "127.0.0.1:5044:5044"
- "127.0.0.1:5601:5601"
- "127.0.0.1:9200:9200"
# TODO preconfigure view via logstash* & @timestamp
jenkins:
build: .
ports:
- "127.0.0.1:8080:8080"
expose:
- "50000"
links:
- elk
agent:
image: cloudbees/jnlp-slave-with-java-build-tools
links:
- elk
- jenkins
# TODO there is no -noreconnect yet this does not work when first started; you need to relaunch it; work around with wait-for-it: https://docs.docker.com/compose/startup-order/
command: -url http://jenkins:8080/ b4178553de10d5551ff6d19cc1f0665d0ee6fff211bc0b57ec7c4c5617e23bb8 remote
73 changes: 55 additions & 18 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,22 @@
<modelVersion>4.0.0</modelVersion>

<properties>
<revision>2.0.0</revision>
<changelist>-SNAPSHOT</changelist>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<skipIntegrationTests>true</skipIntegrationTests>
<jenkins-test-harness.version>2.23</jenkins-test-harness.version>
<java.level>7</java.level>
<jenkins.version>2.7.4</jenkins.version>
<java.level>8</java.level>
<jenkins.version>2.121.1</jenkins.version>
<useBeta>true</useBeta>
</properties>

<parent>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>plugin</artifactId>
<version>2.15</version>
<version>3.14</version>
<relativePath/>
</parent>

<repositories>
<repository>
<id>repo.jenkins-ci.org</id>
Expand All @@ -34,7 +37,7 @@
<connection>scm:git:git://github.com/jenkinsci/logstash-plugin.git</connection>
<developerConnection>scm:git:git@github.com:jenkinsci/logstash-plugin.git</developerConnection>
<url>https://github.com/jenkinsci/logstash-plugin</url>
<tag>HEAD</tag>
<tag>${scmTag}</tag>
</scm>

<distributionManagement>
Expand All @@ -52,7 +55,7 @@

<artifactId>logstash</artifactId>
<packaging>hpi</packaging>
<version>2.0.0-SNAPSHOT</version>
<version>${revision}${changelist}</version>
<name>Logstash</name>
<description>A Logstash agent to send Jenkins logs to a Logstash indexer.</description>
<url>https://plugins.jenkins.io/logstash</url>
Expand Down Expand Up @@ -121,11 +124,54 @@
<artifactId>junit</artifactId>
<version>1.10</version>
</dependency>

<!-- For indexing of jobs in Elasticsearch -->
<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>unique-id</artifactId>
<version>2.1.1</version>
</dependency>

<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-job</artifactId>
<version>2.22-rc303.18d78f305a45</version> <!-- TODO https://github.com/jenkinsci/workflow-job-plugin/pull/27 -->
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-support</artifactId>
<version>2.19-rc257.cd70bcc4c209</version> <!-- TODO https://github.com/jenkinsci/workflow-support-plugin/pull/15 -->
</dependency>
<!-- For testing of Log wrapping in Pipeline -->
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-durable-task-step</artifactId>
<version>2.20-rc333.74dc7c303e6d</version> <!-- TODO https://github.com/jenkinsci/workflow-durable-task-step-plugin/pull/21 -->
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-cps</artifactId>
<version>2.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-basic-steps</artifactId>
<version>2.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>credentials-binding</artifactId>
<version>1.15</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>structs</artifactId>
<version>1.2</version>
<version>1.14</version>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
Expand All @@ -142,7 +188,7 @@
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-step-api</artifactId>
<version>1.15</version>
<version>2.13</version>
<optional>true</optional>
</dependency>
</dependencies>
Expand Down Expand Up @@ -174,15 +220,6 @@
</dependencies>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

<!-- http://jira.codehaus.org/browse/MRELEASE-812 -->
<plugin>
<artifactId>maven-release-plugin</artifactId>
Expand Down
Loading