Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Masterfiles stage rewrite #2465

Merged
merged 35 commits into from Oct 5, 2016
Merged

Masterfiles stage rewrite #2465

Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
18cca43
Change DIRNAME to mydirname as it's not an environment variable.
Jan 5, 2016
3bf4eb9
Fix error_exit calls for consistency
Jan 5, 2016
18f844c
Add comment specifying env vars available
Jan 5, 2016
42da03f
Refactor STAGING_DIR setting to within function calls.
Jan 5, 2016
01e7822
Refactor duplicate code to function set_staging_dir_from_params
Jan 5, 2016
96a849e
Refactor out check_git_installed function
Jan 5, 2016
c016016
Add and incorporate validate_staged_policy function
Jan 6, 2016
6d499c5
Add functions for handling rollout:
Jan 6, 2016
facdcdb
Add function git_setup_local_mirrored_repo
Jan 6, 2016
f11edd9
Add function git_stage_refspec
Jan 6, 2016
5ffc134
Add function git_stage_policy_channels_from_mirror
Jan 6, 2016
ae11575
Add GIT_MIRROR_POLICY_CHANNELS as VCS_TYPE
Jan 6, 2016
b1c4168
Add success message to git_stage_policy_channels_from_mirror
Jan 6, 2016
a8a6a67
Refactor git masterstage funcs to call policy rollout funcs
Jan 6, 2016
51933de
Change git masterstage functions to use mirrored repo
Jan 6, 2016
53edd2d
Add fully rewritten options.sh
Jan 7, 2016
5764aa3
Remove optparse library.
Jan 7, 2016
e85296d
Refactor:
Jan 7, 2016
924da17
Update readme.org:
Jan 7, 2016
a332eb9
Remove 'mirror' from name of VCS_TYPE
Jan 7, 2016
5d9c42c
Created new param, channel_deployment_dir
Jan 7, 2016
00a73f5
Fixes and improvements to git_policy_channels function
Jan 20, 2016
88d595b
Added example params file for git policy channels
Jan 20, 2016
2dda41a
Stop putting any files in 'ROOT'.
Jan 22, 2016
9a1a2ca
Populate .git/HEAD with SHA even if on a git branch.
Jan 22, 2016
ffd7a51
Inlined option parsing into masterfiles-stage.sh
Jan 22, 2016
d7919ea
Add inline documentation of global vars dependencies.
Jan 22, 2016
a1772e1
Refactor - extract function git_deploy_refspec
Jan 22, 2016
362e11b
Consolidate and clean up code, fix Design Center integration:
Jan 22, 2016
d26907f
Add example value that could be set for local_mirrored_repo
Jan 22, 2016
986aa45
Remove obsolete comment line
Jan 22, 2016
537f3a9
Improve permissions handling to prevent warning messages
Jan 22, 2016
19c6ddb
Removed GIT_TAG_OR_COMMIT and document it.
Jan 22, 2016
387b32c
Remove readme reference to options.sh that doesn't now exist.
Jan 22, 2016
a57c700
Add trap handling to common.sh
Jan 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
37 changes: 25 additions & 12 deletions contrib/masterfiles-stage/README.org
View file
Open in desktop
Expand Up @@ -38,23 +38,26 @@ simple environments this will be `$(sys.masterdir)` (typically
- VCS_TYPE="GIT"
- VCS_TYPE="GIT_TAG_OR_COMMIT"
- VCS_TYPE="SVN"
- VCS_TYPE="GIT_POLICY_CHANNELS"

* Dependancies
This script has the following dependancies
* Dependencies
This script has the following dependencies
- bash
- rsync
- rsync (SVN only)
- git|svn

* Installation
Installation simply consists of saving `masterfiles-stage.sh` and its
supporting `common.sh` to the same directory on your hub and making
sure that `masterfiles-stage.sh` is executable.
supporting `common.sh` and `options.sh` to the same directory on your hub
and making sure that `masterfiles-stage.sh` is executable.

#+begin_src example
wget https://raw.githubusercontent.com/cfengine/core/master/contrib/masterfiles-stage/masterfiles-stage.sh -O /var/cfengine/bin/masterfiles-stage.sh
wget https://raw.githubusercontent.com/cfengine/core/master/contrib/masterfiles-stage/common.sh -O /var/cfengine/bin/common.sh
chown root:root /var/cfengine/bin/masterfiles-stage.sh
wget https://raw.githubusercontent.com/cfengine/core/master/contrib/masterfiles-stage/options.sh -O /var/cfengine/bin/options.sh
chown root:root /var/cfengine/bin/masterfiles-stage.sh /var/cfengine/bin/common.sh /var/cfengine/bin/options.sh
chmod 555 /var/cfengine/bin/masterfiles-stage.sh
chmod 444 /var/cfengine/bin/common.sh /var/cfengine/bin/options.sh
#+end_src

*Warning:* The above commands may not work out of the box depending on
Expand All @@ -64,8 +67,9 @@ sure that `masterfiles-stage.sh` is executable.

The `masterfiles-stage.sh` script takes options in order to specify the
directory where the masterfiles should be deployed after staging, as well as
the file that contains the paramaters that should be used in order to interface
the file that contains the parameters that should be used in order to interface
with the given upstream source implementation (git url, branch etc ..)
Run masterfiles-stage.sh --help to see exact option flags.

- `MASTERDIR` is the fully qualified path to the location that you
want masterfiles to be deployed to after successful staging.
Expand All @@ -75,7 +79,7 @@ with the given upstream source implementation (git url, branch etc ..)
staging method (currently reffered to as `VCS_TYPE`). The format of
the file is a simple KEY="VALUE". It is sourced by the
`masterfiles-stage.sh` script. Each staging method or `VCS_TYPE`
defines may define its own parameters. It is reccomended that the
defined may define its own parameters. It is recommended that the
parameters defined by the `VCS_TYPE` be prefixed with the `VCS_TYPE`
so as not to conflict with other staging methods.

Expand All @@ -86,7 +90,7 @@ The following parameters are in use, be sure to avoid their collision
when developing new staging methods.

- VCS_TYPE - The staging method to use
- Currently supported: (GIT|GIT_TAG_OR_COMMIT|SVN)
- Currently supported: (GIT|GIT_TAG_OR_COMMIT|GIT_POLICY_CHANNELS|SVN)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a note here that GIT_TAG_OR_COMMIT is legacy and equivalent to the other one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll hold off on that until Nick and I come to an agreement regarding mirrored repos and how cf_promises_release_id gets populated.


- GIT_URL - The URL to the git repository
- Example values:
Expand All @@ -101,6 +105,15 @@ when developing new staging methods.
- 88335d36b48c8808b12b48667a463182dc8d0338
- mytag

- channel_config_file - The path to the config file for multiple policy channels
- Example value:
- /var/cfengine/policy_channels/channel_to_source.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be done with bash arrays in the params or options instead of a new separate file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea. I think yes, it could. (It would be in params, not options, as it's VCS_TYPE specific.) I think simplest would be just one array channel_config which would be expected to contain at least two elements, and if it had an odd number of elements the last would be ignored (perhaps with a warning).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, yeah. I was actually thinking of bash associative arrays but those may not be available in older shells (bash 4 got these IIRC):

declare -A channels
channels[vcs]="git://..."
...

So any implementation is OK with me, as long as it doesn't require separate config files.


- channel_deployment_dir - The directory in which to deploy each policy channel
(each in a separate subdirectory named according to channel name)
- Example value:
- /var/cfengine/policy_channels/masterfiles_dirs

- SVN_URL - The URL to the svn repository

- SVN_BRANCH - The svn branch to checkout
Expand All @@ -121,22 +134,22 @@ have no effect on functionality of `masterfiles-stage.sh`
operations (not used by `masterfiles-stage.sh`)
- This is used by Design Center in CFEngine Enterprise

- PKEY - Private SSH Key used for authrorization when pushing
- PKEY - Private SSH Key used for authorization when pushing
integrated write operations
- This is used by Design Center in CFEngine Enterprise

- SCRIPT_DIR - Directory contianing supporting scripts
- This is used by Design Center in CFEngine Enterprise

*** Special Note
The executing user may require addational configuration in order to
The executing user may require additional configuration in order to
successfully authenticate. The specifics depend on the specifics of
your upstream repository. For example, an upstream repository over
http or https that allows anonymous access may need no addtional
configuration, but an upstream that uses ssh keys for authentication
may require trust establishment (accepting the host key fingerprint),
and the presence of an ssh key for authentication. Depending on the
security requirements of your orginization, you may use a
security requirements of your organization, you may use a
passphraseless key, or you may use something like ssh-agent.


Expand Down