New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow nesting of templates for inheritance. #94

Merged
merged 1 commit into from Nov 30, 2016

Conversation

Projects
None yet
5 participants
@iocanel

iocanel commented Nov 29, 2016

No description provided.

@carlossg carlossg merged commit 7cec154 into jenkinsci:master Nov 30, 2016

1 check passed

Jenkins This pull request looks good
Details
@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 1, 2017

Does this feature actually work? Cause I can't get it to work as documented.

First the documentation has a few errors (syntax errors). Outside of that I have the following

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

  def defaultLabel = 'maven'
  def label = parameters.get('label', defaultLabel)

  def mavenImage = parameters.get('mavenImage', 'quay.io/xxxx/docker-build-maven:latest')

  podTemplate(label: label,
    containers: [
      containerTemplate(name: 'maven', image: "${mavenImage}", ttyEnabled: true, alwaysPullImage: true, command: 'cat')
    ]) {
    body()
  }
}

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

def defaultLabel = 'docker'
def label = parameters.get('label', defaultLabel)

def dockerImage = parameters.get('dockerImage', 'docker:17.06.0-ce-git')

podTemplate(label: "label",
containers: [
containerTemplate(name: 'docker', image: "${dockerImage}", ttyEnabled: true, alwaysPullImage: true, command: 'cat')
],
volumes: [
hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')
]) {
body()
}
}

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

  def defaultLabel = "maven.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
  def label = parameters.get('label', defaultLabel)

  mavenTemplate(label: label) {
    dockerTemplate(label: label) {
      node(label) {
        body()
      }
    }
  }
}

As you can see I define two templates. Then in my node I combine those. However when the POD is created it only ever gets the container specified in the first POD template.

I put println statements in both template definitions and they execute so the code is running. 

matthewceroni commented Aug 1, 2017

Does this feature actually work? Cause I can't get it to work as documented.

First the documentation has a few errors (syntax errors). Outside of that I have the following

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

  def defaultLabel = 'maven'
  def label = parameters.get('label', defaultLabel)

  def mavenImage = parameters.get('mavenImage', 'quay.io/xxxx/docker-build-maven:latest')

  podTemplate(label: label,
    containers: [
      containerTemplate(name: 'maven', image: "${mavenImage}", ttyEnabled: true, alwaysPullImage: true, command: 'cat')
    ]) {
    body()
  }
}

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

def defaultLabel = 'docker'
def label = parameters.get('label', defaultLabel)

def dockerImage = parameters.get('dockerImage', 'docker:17.06.0-ce-git')

podTemplate(label: "label",
containers: [
containerTemplate(name: 'docker', image: "${dockerImage}", ttyEnabled: true, alwaysPullImage: true, command: 'cat')
],
volumes: [
hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')
]) {
body()
}
}

#!/usr/bin/groovy

def call(Map parameters = [:], body) {

  def defaultLabel = "maven.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
  def label = parameters.get('label', defaultLabel)

  mavenTemplate(label: label) {
    dockerTemplate(label: label) {
      node(label) {
        body()
      }
    }
  }
}

As you can see I define two templates. Then in my node I combine those. However when the POD is created it only ever gets the container specified in the first POD template.

I put println statements in both template definitions and they execute so the code is running. 
@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 2, 2017

@matthewceroni: I recall this feature having some issues in 0.11, but I expect it to work fine in 0.12.

It's something that I've been extensively using the last couple of months (using a build straight from master).

Note: When the templates are internally composed they are looked up by name, so make sure you also add a name to your templates. I think that we need to clarify that in the docs.

iocanel commented Aug 2, 2017

@matthewceroni: I recall this feature having some issues in 0.11, but I expect it to work fine in 0.12.

It's something that I've been extensively using the last couple of months (using a build straight from master).

Note: When the templates are internally composed they are looked up by name, so make sure you also add a name to your templates. I think that we need to clarify that in the docs.

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

By name you mean the label that is set for each podTemplate? Assuming they must match. Or is there another parameter?

matthewceroni commented Aug 2, 2017

By name you mean the label that is set for each podTemplate? Assuming they must match. Or is there another parameter?

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 2, 2017

There is an other string parameter called name that uniquely identifies the pod template.

iocanel commented Aug 2, 2017

There is an other string parameter called name that uniquely identifies the pod template.

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

podTemplate(name: 'mattpod', label: 'docker', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
    podTemplate(name: 'mattpod', label: 'docker', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
      node('docker') {
        stage ('Testing') {
          sh "echo here"
          sh "/bin/sleep 600"
        }
      }
    }
  }

This is what I am trying now and the POD get created with only two containers (the jnlp and docker). The maven container doesn't get included. I am on the latest release so will try using a build of master. 

Assuming my syntax is correct above. 

matthewceroni commented Aug 2, 2017

podTemplate(name: 'mattpod', label: 'docker', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
    podTemplate(name: 'mattpod', label: 'docker', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
      node('docker') {
        stage ('Testing') {
          sh "echo here"
          sh "/bin/sleep 600"
        }
      }
    }
  }

This is what I am trying now and the POD get created with only two containers (the jnlp and docker). The maven container doesn't get included. I am on the latest release so will try using a build of master. 

Assuming my syntax is correct above. 
@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

So I installed the latest plugin from master (version 0.13-SNAPSHOT)

Then I tested the following PIPELINE

podTemplate(name: 'mypod', label: 'mypod', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     podTemplate(name: 'mypod', label: 'mypod2', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     node('mypod') {
         stage('Run shell') {
             sh 'echo hello world'
         }
     }
     }
 }

But same issue as before. The POD gets created by only the docker container is present (plus the default JNLP). The maven container does not get included.

From the PIPELINE above can you see any issue? I set the name to match and even tried having the label match or not match.

matthewceroni commented Aug 2, 2017

So I installed the latest plugin from master (version 0.13-SNAPSHOT)

Then I tested the following PIPELINE

podTemplate(name: 'mypod', label: 'mypod', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     podTemplate(name: 'mypod', label: 'mypod2', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     node('mypod') {
         stage('Run shell') {
             sh 'echo hello world'
         }
     }
     }
 }

But same issue as before. The POD gets created by only the docker container is present (plus the default JNLP). The maven container does not get included.

From the PIPELINE above can you see any issue? I set the name to match and even tried having the label match or not match.

@rawlingsj

This comment has been minimized.

Show comment
Hide comment
@rawlingsj

rawlingsj Aug 2, 2017

Just a guess but it maybe the containerTemplate you have above. I.e. here's a working example that we use..
https://github.com/fabric8io/fabric8-pipeline-library/blob/41f9f4a/vars/mavenTemplate.groovy#L19-L43

rawlingsj commented Aug 2, 2017

Just a guess but it maybe the containerTemplate you have above. I.e. here's a working example that we use..
https://github.com/fabric8io/fabric8-pipeline-library/blob/41f9f4a/vars/mavenTemplate.groovy#L19-L43

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

@rawlingsj

I have actually looked at the example you pasted. But that isn't using the POD template nesting feature.

matthewceroni commented Aug 2, 2017

@rawlingsj

I have actually looked at the example you pasted. But that isn't using the POD template nesting feature.

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 2, 2017

@matthewceroni: you can't use the same name on both templates. Use unique names and you should be fine.

iocanel commented Aug 2, 2017

@matthewceroni: you can't use the same name on both templates. Use unique names and you should be fine.

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

@iocanel

The podTemplate(name: .. ) settings needs to differ?

podTemplate(name: 'mypod', label: 'mypod', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     podTemplate(name: 'mypod2', label: 'mypod', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     node('mypod') {
         stage('Run shell') {
             sh 'echo hello world'
         }
     }
     }
 }

Like that? Cause I did that and still doesn't work.

matthewceroni commented Aug 2, 2017

@iocanel

The podTemplate(name: .. ) settings needs to differ?

podTemplate(name: 'mypod', label: 'mypod', containers: [containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     podTemplate(name: 'mypod2', label: 'mypod', containers: [containerTemplate(name: 'maven', image: 'maven', ttyEnabled: true, alwaysPullImage: true, command: 'cat')]) {
     node('mypod') {
         stage('Run shell') {
             sh 'echo hello world'
         }
     }
     }
 }

Like that? Cause I did that and still doesn't work.

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 2, 2017

@matthewceroni: Yeah, this seems right and it should work 0.12 onwards.
As @rawlingsj commented above, this is something extensively used in the fabric8 pipeline library. In the linked example the mavenTemplate is nestable.

iocanel commented Aug 2, 2017

@matthewceroni: Yeah, this seems right and it should work 0.12 onwards.
As @rawlingsj commented above, this is something extensively used in the fabric8 pipeline library. In the linked example the mavenTemplate is nestable.

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 2, 2017

Can you try and also use a unique value for the label too?

iocanel commented Aug 2, 2017

Can you try and also use a unique value for the label too?

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 2, 2017

@iocanel I believe I have tried using a unique label but will try again.

In the mavenTemplate example they are over-writing the maven container. As the nested (possible) podTemplate contains the same maven container just with different settings.

I am trying to add additional containers. Is that possible or is the nesting feature only able to over-write containers that are in the outter podTemplate.

matthewceroni commented Aug 2, 2017

@iocanel I believe I have tried using a unique label but will try again.

In the mavenTemplate example they are over-writing the maven container. As the nested (possible) podTemplate contains the same maven container just with different settings.

I am trying to add additional containers. Is that possible or is the nesting feature only able to over-write containers that are in the outter podTemplate.

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Aug 3, 2017

It's possible to both add and replace.

More real world examples:

  • The Kubernetes Client
    Jenkinsfile
    which is using the following templates nested:
    -- clientsTemplate adds clients template (oc, kubectl binaries)
    -- mavenNode adds maven template and also wraps inside a node.

  • Syndesis Rest
    Jenkinsfile
    which is using the following templates nested:
    -- withMaven adds maven templates
    -- withOpenshift adds openshift template
    -- slave customizes the jnlp

In both examples the idea is the same, use simple functions to define pod templates, and blend them together using nesting.

iocanel commented Aug 3, 2017

It's possible to both add and replace.

More real world examples:

  • The Kubernetes Client
    Jenkinsfile
    which is using the following templates nested:
    -- clientsTemplate adds clients template (oc, kubectl binaries)
    -- mavenNode adds maven template and also wraps inside a node.

  • Syndesis Rest
    Jenkinsfile
    which is using the following templates nested:
    -- withMaven adds maven templates
    -- withOpenshift adds openshift template
    -- slave customizes the jnlp

In both examples the idea is the same, use simple functions to define pod templates, and blend them together using nesting.

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 3, 2017

The Syndesis Rest examples make sense to me. So I tried to follow them.

https://github.com/Smarsh/pipeline-library/tree/master/vars

And my test Jenkinsfile

 @Library('pipeline-library@master') _  
 
 mavenTemplate {
    dockerTemplate {
      node('maven') {
        stage('testing') {
            sh "/bin/sleep 600"
        }
      }  
    }
  }

Still nothing. This is frustrating me since it seems so straight forward.

What I do notice is that if you set the name on podTemplate the POD it creates starts with that name (ie: maven-######).

The only thing I don't have that the above examples have is setting the defaultLabel (ex: def defaultLabel = buildId('maven')) as function buildId is undefined but I don't think that should really matter.

matthewceroni commented Aug 3, 2017

The Syndesis Rest examples make sense to me. So I tried to follow them.

https://github.com/Smarsh/pipeline-library/tree/master/vars

And my test Jenkinsfile

 @Library('pipeline-library@master') _  
 
 mavenTemplate {
    dockerTemplate {
      node('maven') {
        stage('testing') {
            sh "/bin/sleep 600"
        }
      }  
    }
  }

Still nothing. This is frustrating me since it seems so straight forward.

What I do notice is that if you set the name on podTemplate the POD it creates starts with that name (ie: maven-######).

The only thing I don't have that the above examples have is setting the defaultLabel (ex: def defaultLabel = buildId('maven')) as function buildId is undefined but I don't think that should really matter.

@matthewceroni

This comment has been minimized.

Show comment
Hide comment
@matthewceroni

matthewceroni Aug 3, 2017

Ok, I think I just got it to work.

The maven podTemplate has default label of maven. The docker podTemplate has a default label of docker.

As you can see in the above I set node('maven') and in doing so it only provisioned the maven podTemplate. If i set that to node('docker') it worked.

My understanding of the parameter to node was to match the label the slave had so I always tried to make the label match on all the podTemplate. But looking at the examples that isn't required.

In the Syndesis example they wrapped their calls to their template in node { } but if I do that the pod never gets provisioned and the PIPELINE just sits there with "Waiting for next available executor"

matthewceroni commented Aug 3, 2017

Ok, I think I just got it to work.

The maven podTemplate has default label of maven. The docker podTemplate has a default label of docker.

As you can see in the above I set node('maven') and in doing so it only provisioned the maven podTemplate. If i set that to node('docker') it worked.

My understanding of the parameter to node was to match the label the slave had so I always tried to make the label match on all the podTemplate. But looking at the examples that isn't required.

In the Syndesis example they wrapped their calls to their template in node { } but if I do that the pod never gets provisioned and the PIPELINE just sits there with "Waiting for next available executor"

@cyrus-mc

This comment has been minimized.

Show comment
Hide comment
@cyrus-mc

cyrus-mc Oct 25, 2017

Is this functionality supported when using declarative pipeline? If so does anyone have an example?

cyrus-mc commented Oct 25, 2017

Is this functionality supported when using declarative pipeline? If so does anyone have an example?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment