Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
4e2a2b8
Initial Commit
npwalker Jan 14, 2015
a6ef384
Add the README
npwalker Jan 14, 2015
f4393ec
Add stdlib to Puppetfile
npwalker Jan 14, 2015
0c67040
Update README.md
npwalker Jan 14, 2015
e2b5da1
Update README.md
npwalker Jan 15, 2015
4fa1454
Update site.pp to work OOTB
npwalker May 11, 2015
2567745
Update hiera.yaml
npwalker May 14, 2015
368cb4a
Update configure_r10k.pp for PE3.8
npwalker May 15, 2015
d3ee945
Update hiera.yaml
npwalker May 15, 2015
2226735
Rename defaults.yaml to common.yaml
npwalker May 15, 2015
52629f3
Update Puppetfile
npwalker May 15, 2015
724e3e9
Update README.md
npwalker May 15, 2015
bfed08b
Update README.md
npwalker May 26, 2015
944a030
add site to modulepath in environment.conf
npwalker May 27, 2015
24aa475
Add a site directory with example role and profile module
npwalker May 27, 2015
4e8e16c
Remove purgedirs
npwalker Jun 25, 2015
e972baf
Update Puppetfile
npwalker Jun 26, 2015
bc6bae2
Update environment.conf
npwalker Jun 26, 2015
437433f
Remove hiera.yaml and instead manage it with puppet code
npwalker Aug 3, 2015
f5cb1fa
Update config_version to use a script that does not require git
npwalker Aug 4, 2015
921c837
Update site.pp to use a trusted fact role
npwalker Aug 4, 2015
5d0ae0c
fix config_version script for 3.8 and below
npwalker Aug 4, 2015
9b50d5e
Update the README, add an all_in_one_pe role
npwalker Aug 13, 2015
b0c5ac4
Add execute bit to config_version scripts
npwalker Aug 13, 2015
7069033
Fix some quoting issues and add role:: to the include in site.pp
npwalker Aug 13, 2015
6a78949
Fix some incorrect quoting in the hiera class
npwalker Aug 14, 2015
6425d2b
Remove extraneous slash from puppetmaster profile
npwalker Aug 14, 2015
2e01811
Add puppet-master.yaml to hieradata
npwalker Aug 14, 2015
ab67926
Add a gitlab profile / add ssh-keygen to puppetmaster profile
npwalker Aug 14, 2015
0bf10ae
Add a gitlab role
npwalker Aug 14, 2015
96fc614
Add vhsn/gitlab and acidprime/r10k to Puppetfile
npwalker Aug 15, 2015
fc3ed99
Add abrader-gms module to Puppetfile
npwalker Aug 17, 2015
1525bf5
Add webhook config to git management system
npwalker Aug 17, 2015
184cf7e
Change default gitlab url to be https instead of http
npwalker Aug 17, 2015
246626e
Copy agent certs into Gitlab directory for Gitlab SSL
npwalker Aug 18, 2015
0af9bbf
Move some hieradata around for easier testing in vagrant
npwalker Aug 18, 2015
1eab000
Update the README with more specific instructions
npwalker Aug 18, 2015
308e73a
Update version of abrader-gms
npwalker Aug 18, 2015
4e555b2
Update README
npwalker Aug 18, 2015
01baf76
Update the repo name for deploy keys and webhook to match README
npwalker Aug 18, 2015
04f3239
add modules for stash and a profile
npwalker Aug 25, 2015
809f682
Add hiera key for tuning max_requests_per_instance
npwalker Sep 22, 2015
06f2e5e
Correct module namespace for stash module
npwalker Sep 22, 2015
fe07a85
Update README partially
npwalker Sep 22, 2015
0c486e2
Ensure /etc/gitlab/ssl exists before putting certs in it
npwalker Sep 24, 2015
03d6c0c
Update README some more
npwalker Sep 24, 2015
c54b900
Update README to remove --mirror from git clone
npwalker Sep 25, 2015
a8cbec5
update README
npwalker Sep 29, 2015
03bfb39
Change max_requests_per_instance to 10K
npwalker Oct 6, 2015
6ff5658
Add some notifies to pe services after making changes
npwalker Oct 15, 2015
1828e21
Change to using the webhook without mcollective
npwalker Oct 16, 2015
683a220
Change puppetdb heap size to 96MB from 64MB
npwalker Oct 16, 2015
6582067
Change the zack/r10k webhook to not authenticate
npwalker Oct 21, 2015
726052f
Remove notes about making control-repo public
npwalker Oct 21, 2015
ae3b7c7
fix whitespace
npwalker Oct 21, 2015
2ffc83a
Clean up the README so it works from top to bottom
npwalker Oct 21, 2015
9d1b750
fix site.pp to not include a role that is empty
npwalker Oct 22, 2015
86cb66b
fix incorrect curly brace
npwalker Oct 22, 2015
ac0f1b0
Multiple fixes to the README
npwalker Oct 22, 2015
baf376d
add .gitignore to ignore my example hiera file
npwalker Oct 22, 2015
92f8d2d
Configure the MoM to update the classifier after deploying code
npwalker Oct 26, 2015
1cb89d9
Fix some embarassing typos
npwalker Oct 26, 2015
4598483
Fix the update-classes script and set appropriate perms on it
npwalker Oct 26, 2015
8fe670b
make environment timeout setting refresh the puppet master service
npwalker Oct 26, 2015
d6cb170
enable ssl on the webhook
npwalker Oct 28, 2015
c2aeccd
disable SSL on the zack/r10k webhook
npwalker Oct 28, 2015
591f9b9
remove environment_timeout =0 from environment.conf
npwalker Oct 28, 2015
b6351f9
Add $::fqdn to post receive hook title
npwalker Oct 28, 2015
e045f17
enable SSL on the zack/r10k webhook
npwalker Oct 29, 2015
0b7aceb
switch to npwalker/abrader-gms
npwalker Oct 29, 2015
3fcf6e3
change puppetserver heap to 192m
npwalker Oct 29, 2015
8e271e3
Change the zack/r10k webhook to utilize username and password
npwalker Oct 30, 2015
3149d97
Refactor webhook profiles into one profile with a parameter
npwalker Oct 30, 2015
e5be3e2
fixing a bunch of puppet lint warnings
npwalker Oct 30, 2015
7e40513
Change owner/group to pe-puppet for hiera module
npwalker Nov 17, 2015
afc9a35
Merge branch 'production' of https://github.com/npwalker/control-repo…
npwalker Nov 24, 2015
1fd5334
Cleanup README to generalize urls and add notes about testing
npwalker Nov 24, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
hieradata/nodes/example-puppet-master.yaml
26 changes: 17 additions & 9 deletions Puppetfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ forge "http://forge.puppetlabs.com"
mod "puppetlabs/inifile", :latest
mod "puppetlabs/stdlib", :latest
mod "puppetlabs/concat", :latest
mod "puppetlabs/ntp", :latest
mod "saz/ssh", :latest
mod "puppetlabs/postgresql", '4.5.0'
mod "puppet/stash", '1.3.0'
mod "puppetlabs/java", '1.4.1'

#An example of using a specific forge module version instead of latest
#Notice the addition of single quotes
Expand All @@ -14,15 +19,18 @@ mod "puppetlabs/concat", :latest
# Modules from Github using various references
# Further examples: https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd#examples
# update the tag to the most current release when implementing
#mod 'hiera',
# :git => 'https://github.com/hunner/puppet-hiera',
# :tag => '1.3.1'
mod 'hiera',
:git => 'https://github.com/hunner/puppet-hiera',
:tag => '1.3.1'

#mod 'notifyme',
# :git => 'git://github.com/glarizza/puppet-notifyme',
# :ref => '50c01703b2e3e352520a9a2271ea4947fe17a51f'
mod 'gitlab',
:git => 'https://github.com/vshn/puppet-gitlab',
:ref => '00397b86dfb3487d9df768cbd3698d362132b5bf'

#mod 'profiles',
# :git => 'git://github.com/glarizza/puppet-profiles',
# :ref => '3611ae4253ff01762f9bda1d93620edf8f9a3b22'
mod 'r10k',
:git => 'https://github.com/acidprime/r10k',
:tag => 'v3.1.1'

mod 'gms',
:git => 'https://github.com/npwalker/abrader-gms',
:branch => 'gitlab_disable_ssl_verify_support'
180 changes: 178 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,178 @@
# control-repo
A control repository template
# Before Starting:

This control repo and the steps below are intended to be used during a new installation of PE.

This control repo has only been tested against PE2015.2.z, it's likely close to working on PE3.8.z but has not been tested.

If you intend to use it on an existing installation then be warned that if you've already written or downloaded modules when you start using r10k it will remove all of the existing modules and replace them with what you define in your Puppetfile. Please copy or move your existing modules to another directory to ensure you do not lose any work you've already started.

## Setup a Trusted Fact On Your PE Master

This control repository is setup to manage certain portions of your PE installation for you if you create a trusted fact called `pp_role`. In order to do so, lay down a file that looks exactly like the below in `/etc/puppetlabs/puppet/csr_attributes.yaml`

```
---
extension_requests:
#pp_role
1.3.6.1.4.1.34380.1.1.13: 'all_in_one_pe'
```

### If You Have Not Installed PE

Good then you can proceed forward and the trusted fact will be used when you get to the install step.

### If You Have Already Installed PE

Trusted facts are created at the time a CSR is generated. So, we need to regenerate the certificate on the master for the above trusted fact to be created.

Follow this document to regenerate the certificate on your master.

http://docs.puppetlabs.com/pe/latest/regenerate_certs_master.html

##Copy This Repo Into Your Own Git Server

###Gitlab

1. Install Gitlab
- https://about.gitlab.com/downloads/

2. After Gitlab is installed you may sign if with the `root` user and password `5iveL!fe`

3. Make an user for yourself

4. Make an ssh key to link with your user. You’ll want to do this on the machine you intend to edit code from ( most likely not your puppet master but your local workstation / laptop )
- http://doc.gitlab.com/ce/ssh/README.html
- https://help.github.com/articles/generating-ssh-keys/

5. Create a group called `puppet` ( this is case sensitive )
- http://doc.gitlab.com/ce/workflow/groups.html

6. Create a user called `r10k_api_user` and add them to the `puppet` group
- From the landing page, select groups
- Choose the puppet group
- In the left hand pane, select memembers
- Add the `r10k_api_user` with `master` permissions

7. Add your user to the `puppet` group as well

8. Create a project called `control-repo` and set the Namespace to be the `puppet` group

9. Logout of root and login as the `r10k_api_user`
- Go to profile settings -> account ( https://<your_gitlab_server>/profile/account )
- Copy the api token

10. Clone this control repository to your laptop/workstation
- `git clone <repository url>`
- `cd control-repo`

11. `git mv hieradata/nodes/example-puppet-master.yaml hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`
- Open `hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`
- edit `gms_api_token` to be your api token
- edit `git_management_system` to be 'gitlab'
- edit the `gms_server_url`

12. `git add hieradata/nodes/<fqdn_of_your_puppet_master>.yaml`

13. `git commit -m "renaming example-puppet-master.yaml"`

14. Rename my repository as the upstream remote
- `git remote rename origin upstream`

15. Add your internal repository as the origin remote
- `git remote add origin <url of your gitlab repository>`

16. Push the production branch of the repository from your machine up to your git server
- `git push origin production`

17. `git branch --set-upstream-to origin/production`

###Stash

Coming soon!

###Github

Coming soon!

##Configure PE to Use the Control-Repo

###Install PE

1. Download the latest version of the PE installer for your platform and copy it to your master
- https://puppetlabs.com/download-puppet-enterprise
2. Expand the tarball and `cd` into the directory
3. Run `puppet-enterprise-installer` to install

If you run into any issues or have more questions about the installer you can see our docs here:

http://docs.puppetlabs.com/pe/latest/install_basic.html

###Get the Control-Repo Deployed On Your Master

At this point you have my control-repo code deployed into your git server. However, we have one final challenge getting that code onto your puppet master. In the end state the master will pull code from the git server via r10k, however, at this moment your puppet master doesn't have credentials to get code from the git server.

So, we'll set up a deploy key in the git server that will allow a ssh-key we make to deploy the code and configure everything else.

1. On your puppet master, make an ssh key for r10k to connect to gitlab
- `/usr/bin/ssh-keygen -t rsa -b 2048 -C 'r10k' -f /root/.ssh/r10k_rsa -q -N ''`
- http://doc.gitlab.com/ce/ssh/README.html
- https://help.github.com/articles/generating-ssh-keys/
2. Create a deploy key on the `control-repo` project in Gitlab
- Paste in the public key from above
- `cat /root/.ssh/r10k_rsa.pub`
3. Follow https://docs.puppetlabs.com/pe/latest/r10k_config_console.html
- The remote is on the front page of the project in the gitlab UI
- git_settings should be:
- `{"provider": "rugged",
"private_key": "/root/.ssh/r10k_rsa"}`
3. Run `puppet agent -t`
- Expect to see changes to `r10k.yaml`
3. Run `r10k deploy environment -pv`
4. Run `puppet agent -t`
- Now you should see many more changes


## Test The Zack/r10k Webhook

One of the components setup by this control-repo is that when you "push" code to your git server, the git server will inform the puppet master to run `r10k deploy environment -p`.

1. Edit README.md
- Just add something to it
2. `git add README.md`
3. `git commit -m "edit README"`
4. `git push origin production`
5. Allow the push to complete and then give it few seconds to complete
- Open `/etc/puppetlabs/code/environments/production/README.md` and confirm your change is present



----
#Miscellaneous

## If You Want to Install Pointing To This Repo on Github

### Setting Up Gitlab

1. Install Gitlab on a server by specifying the following trusted fact on the soon-to-be Gitlab server and then [install the PE agent](http://docs.puppetlabs.com/pe/latest/install_agents.html#using-the-puppet-agent-package-installation-script).

```
---
extension_requests:
#pp_role
1.3.6.1.4.1.34380.1.1.13: 'gitlab'
```

### Setting up Github

Not yet completed.

### Setting up Stash

Not yet completed.


#TODO
Flush out generating an answer file and then appending extra answers onto the end of it.



6 changes: 0 additions & 6 deletions environment.conf
Original file line number Diff line number Diff line change
@@ -1,8 +1,2 @@
modulepath = modules:site:$basemodulepath
config_version = 'scripts/config_version.sh $environmentpath $environment'

# Environment timeout should be set to unlimited. When set to zero it is less performant.
# When code is deployed the admin API of puppetserver should be used to force a refresh of code from disk.
# https://docs.puppetlabs.com/puppetserver/1.0/release_notes.html#new-feature-admin-api-for-refreshing-environments
# https://docs.puppetlabs.com/puppet/3.7/reference/environments_configuring.html#environmenttimeout
environment_timeout = 0
8 changes: 8 additions & 0 deletions hieradata/common.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
---
message: "This node is using common data"

#Puppet Server Tuning
puppet_enterprise::master::puppetserver::jruby_max_requests_per_instance: 10000

#pe-console-services tuning
#https://docs.puppetlabs.com/pe/latest/console_config.html#tuning-the-classifier-synchronization-period
#disable classifier scheduled sync and rely on r10k postrun command to sync the classes
puppet_enterprise::profile::console::classifier_synchronization_period: 0
8 changes: 8 additions & 0 deletions hieradata/nodes/example-puppet-master.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
git_management_system: 'gitlab'
gms_server_url: 'https://gitlab-server'
gms_api_token: 'BDkZfWWnk4LVTLHdAywd'

#setup r10k to update classes in the console after code deploy
pe_r10k::postrun:
- '/usr/local/bin/update-classes.sh'
42 changes: 42 additions & 0 deletions hieradata/virtual/virtualbox.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#These setting are intended for low memory testing VMs
#Not intended for general usage
---
#Drop puppetdb Java Heap Size
#PE3.2 and above
pe_puppetdb::pe::java_args:
-Xmx: '256m'
-Xms: '64m'
#PE3.1 and below
pe_puppetdb::java_args:
-Xmx: '256m'
-Xms: '64m'
#Drop the activemq java heap size
pe_mcollective::role::master::activemq_heap_mb: '96'
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
pe_puppetdb::pe::listen_address: '0.0.0.0'

#PE3.7+
#Allow access to the puppetdb performance dashboard from non-localhost
#This is insecure and also allows access to all API endpoints without verification
puppet_enterprise::profile::puppetdb::listen_address: '0.0.0.0'
puppet_enterprise::profile::amq::broker::heap_mb: '96'
puppet_enterprise::profile::master::java_args:
Xmx: '192m'
Xms: '128m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::puppetdb::java_args:
Xmx: '96m'
Xms: '96m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::profile::console::java_args:
Xmx: '64m'
Xms: '64m'
'XX:MaxPermSize': '=96m'
'XX:PermSize': '=64m'
puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1 #PE3.7.2 only
puppet_enterprise::profile::console::delayed_job_workers: 1
#shared_buffers takes affect during install but is not managed after
puppet_enterprise::profile::database::shared_buffers: '4MB'
12 changes: 9 additions & 3 deletions manifests/site.pp
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@
# Define filebucket 'main':
filebucket { 'main':
#server should point to one master that will be the file bucket
#$servername is the closest thing to the current behavior of PE
#which hardcodes each master to point to itself for a file buckket
server => $servername,
server => "${settings::server}",
path => false,
}

Expand All @@ -39,6 +37,14 @@
# specified in the console for that node.

node default {
#incude a role on any node that specifies it's role via a trusted fact at provision time
#https://docs.puppetlabs.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#trusted-facts
#https://docs.puppetlabs.com/puppet/latest/reference/ssl_attributes_extensions.html#aws-attributes-and-extensions-population-example

if !empty( $trusted['extensions']['pp_role'] ) {
include "role::${trusted['extensions']['pp_role']}"
}

# This is where you can declare classes for all nodes.
# Example:
# class { 'my_class': }
Expand Down
2 changes: 1 addition & 1 deletion scripts/config_version.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@

#add something to find the remote url

puts head_sha
puts head_sha
2 changes: 1 addition & 1 deletion scripts/config_version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ then
/opt/puppetlabs/puppet/bin/ruby $1/$2/scripts/config_version.rb $1 $2
else
/usr/bin/git --git-dir $1/$2/.git rev-parse HEAD
fi
fi
11 changes: 11 additions & 0 deletions site/profile/files/puppetmaster/update-classes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#https://docs.puppetlabs.com/pe/latest/nc_update_classes.html#post-v1update-classes

CONFDIR=$(puppet master --configprint confdir)

CERT=$(puppet master --confdir ${CONFDIR} --configprint hostcert)
CACERT=$(puppet master --confdir ${CONFDIR} --configprint localcacert)
PRVKEY=$(puppet master --confdir ${CONFDIR} --configprint hostprivkey)
OPTIONS="--cert ${CERT} --cacert ${CACERT} --key ${PRVKEY}"
CONSOLE=$(awk '/server: /{print $NF}' ${CONFDIR}/classifier.yaml)

curl -k -X POST ${OPTIONS} "https://${CONSOLE}:4433/classifier-api/v1/update-classes"
5 changes: 5 additions & 0 deletions site/profile/manifests/base.pp
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
class profile::base {

#the base profile should include component modules that will be on all nodes

}
24 changes: 24 additions & 0 deletions site/profile/manifests/gitlab.pp
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
class profile::gitlab {

file { ['/etc/gitlab', '/etc/gitlab/ssl'] :
ensure => directory,
}

file { "/etc/gitlab/ssl/${::fqdn}.key" :
ensure => file,
source => "${::settings::privatekeydir}/${::trusted['certname']}.pem",
notify => Exec['gitlab_reconfigure'],
}

file { "/etc/gitlab/ssl/${::fqdn}.crt" :
ensure => file,
source => "${::settings::certdir}/${::trusted['certname']}.pem",
notify => Exec['gitlab_reconfigure'],
}

class { 'gitlab':
external_url => hiera( 'gms_server_url', "https://${::fqdn}") ,
require => File["/etc/gitlab/ssl/${::fqdn}.key", "/etc/gitlab/ssl/${::fqdn}.key"],
}

}
Loading