Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Consider separating cookbooks into separate repos #165

Open
wants to merge 1 commit into from

6 participants

@patcon

Hey guys,

How would you feel about separating cookbooks into different repositories? This is actually the official convention for Chef now. We could start a new org called "travis-cookbooks" to help keep it organized. We could also pull the cookbooks back into the main repo (the container repo that replaces this one) using librarian (or alternatively berkshelf, if anyone feels strongly that its the better option).

Benefits

  • easier means of sharing cookbook with community, and swapping custom-baked travis cookbooks for community-supported cookbooks
  • easier for user to re-use cookbooks. librarian makes it possible to pull individual cookbooks out of the current repo, but it involves cloning the whole 100MB repo just to get --say-- the xserver cookbook.
  • more likely that those outside the travis project will contribute to cookbooks
@patcon

github.com/applicationsonline/librarian
https://github.com/RiotGames/berkshelf

@michaelklishin

I don't think we have all those problems. Cookbooks that can be standalone are sometimes pulled out into separate repos (under my own account). Travis cookbooks are often not useful in other environments, so there is no upside
to moving some 80+ cookbooks into 80+ repos.

@michaelklishin

I also don't see how having all cookbooks together is hard for contributors. With Sous Chef I'd argue the opposite is true.

@joshk joshk reopened this
@joshk
Owner

I would like to give this 5mins.

I see some merit in this idea, but also see some disadvantages.

The good thing is that most of our cookbooks don't change often, and it does make it easier to share our recipes.

Leaving this open for now so we can all think about it a bit more :)

@michaelklishin

Few cookbooks in travis-cookbooks are both valuable and generic enough:

  • Neo4J Server
  • Cassandra
  • GHC
  • Go/GVM
  • Gradle

and maybe a couple more. They have been extracted long time ago. Everything else is either very Travis-specific (e.g. PHP or Python, RVM, MySQL, PostgreSQL) or is synced with upstream (Riak, RabbitMQ) or does not
change beyond version bumps (MongoDB, Leiningen).

Keeping track of changes across 80+ repositories sounds like a lot of effort for maintainers at effectively no benefit (for maintainers). Sorry, this is only a good idea in theory.

@patcon

You're definitely right about it being overkill to split out every cookbook @michaelklishin -- I'm totally with you on that. I'm just thinking that pulling in the canonical opscode cookbooks into cookbooks-external would be helpful in managing. It will help ensure that worker_host and ci_environment are using some common cookbooks that are covered in tests. (also becomes more DRY)

Also, leveraging community cookbooks and keeping up to date means that you'll get lots more cross-platform support for free if/when travis starts to support multiple operating systems.

And I know it's not technically a benefit to the maintainers if they're pushing travis cookbooks out into the Chef community, but I think making the pieces more decomposable and consumable will result in more people taking interest in your cookbooks, and more eyes on open source code is always better, right?

I've started pulling out some of the community cookbooks, just to try to show what this could look like. I'm having to make some assumptions in moving to the canonical opscode cookbooks (like that having an extra unused recipe in ci_worker isn't bad, or that the tweaked build-essential cookbook could instead install zsh in a separate cookbook). I'm trying to make the commits atomic so it's easy to see what I've done. I'm also making corresponding changes to travis-boxes, and I'll push that later.

I don't mind if I do some of this and it's a dead-end, but def let me know if you have any thoughts :)
https://github.com/patcon/travis-cookbooks/compare/librarian-support

@patcon

Oh hey, sorry if this is on your radar already, but sous-chef could use this plugin and to make sure the cookbooks/ dir is always present:
https://github.com/thegcat/vagrant-librarian

@gildegoma
Collaborator

Definitely :+1: !

I've been a very happy user of librarian for a quite a while, and I've recently started to switch to Berkshelf (mostly because of nice Vagrant integration, Opscode involvement strong trend, Community activity). These cookbook bundlers can combine very well with Sous-Chef logic to reach DRY principle and smooth upgrades on standard stuff (think about generic things like 'apt' or even 'postgresql', where the community cookbooks are very mature and flexible). As an example (that I know ;-), the "sbt" cookbook in travis-cookbooks can be integrated as an exact copy of http://community.opscode.com/cookbooks/sbt-extras, only the values in attributes/default.rb must be overridden, which is easy to achieve with these cookbook bundlers and chef-solo. Beside attributes, we also sometimes need to override templates, but that's not a big deal.

About the split effort, why not select 2-3 of the toughest Travis cookbooks, and try in a feature branch to extract the travis customization, and integrate them with related community cookbooks ? I feel that it won't be so huge work...

Of course, I know the path would not be that easy (and regression risky), but I'm convinced it is worth it in long term perspective. I'm feeling ready to contribute at maximum if core team decides to go forward in this direction.

@gildegoma
Collaborator

@patcon thanks for linking the vagrant-librarian plugin, if librarian main project integrates it in its core, it would one advantage less in favor of http://berkshelf.com. That would be great if librarian and berkshelf start a merge process, as Jamie CI and Test-Kitchen did.

@patcon

Thanks for the input @gildegoma :)
Re: berkshelf-librarian collaboration: berkshelf/berkshelf#396

@patcon

if librarian main project integrates it in its core, it would one advantage less

I'm actually in favour of unix-y projects that try to do less things but remain extensible

@gildegoma
Collaborator

@patcon totally agreed about modularity, KISS and "small is beautiful" unixness. "core" was really a bad wording in this case, sorry :worried:.
My concern was actually more about ensuring that the plugin gain community effect and evolves at the same pace than the rest of librarian toolkit (hum, let's say that some day the github repo should move to something like applicationsonline/vagrant-librarian ;-). We'll see...

If we would start a small cookbook-bundler PoC "now", I would give a small favor to Berkshelf, because it is the cookbook bundler integrated by default in Test-Kitchen 1.0. And I feel that we should also take advantage of Test-Kitchen to run regression testing against different kind of VMs (bluebox linux 64bit / 32bit, mac os x,...).

But focusing back on the main question: I think that all Travis customizations should still be stored in a single repository (travis-cookbooks, always!), but the great improvement would be to extract the standard/community stuff.

@patcon

Ah... gotcha! Yeah, github organizations are great for that sort of thing :)

+1 test-kitchen! I'm down with doing this PoC with berkshelf, as I think the syntax is essentially identical

@gildegoma
Collaborator

4 new requests related to Berkshelf or Librarian integration were recently created (#170, #176, #181 and #182).

At the moment, I think that such requests don't make sense from Travis VM Provisioning perspective (Chef-Solo, Sous Chef, travis_build_environment attributes used from any recipes across the single repository). So far, contributors to Travis project don't need to use any cookbook bundlers, because travis-cookbooks is "standalone".

My summary of the situation:

  • At the moment, Travis Cookbooks organization is not ready to be used a "generic way" (hence the current issue being discussed).
  • Cookbook integration in https://github.com/travis-ci/travis-cookbooks is (and must remain) only focused on Travis machines.
  • As Michael already mentioned above, some generic cookbooks integrated in travis-cookbooks are already available in separate repositories (without travis_build_environment dependencies).
  • I don't think It is a good idea to partially 'patch' inside travis-cookbooks to make Berkshelf or Librarian 'happy'. If there is "generic" interest on a specific travis cookbook (that has not been extracted yet), the community should extract and generalize it.

To @michaelklishin, @roidrage, @joshk, etc.: If you agree with my points above, I'd propose to add a note in https://github.com/travis-ci/travis-cookbooks/blob/master/README.md, to warn that these cookbooks are not intended to be directly bundled from this repository (maybe: +links to available generic cookbooks). What do you think?

Question to @hiremaga, @drnic, @MasseGuillaume, @cf-bosh: if possible, could you please shortly describe your integration context (which cookbooks?). @patcon and I are interested in this "costs and benefits of reusability", do you want to join and start to work on a cookbook generalization?

@drnic
@gildegoma
Collaborator

thanks @drnic for feedback. It is good to know your context. First, to make it quite clear: it must be open source!

Sous-Chef and Vagrant is enough to provision a Travis VM. You need more formal metatdata.rb because of Berkshelf, and you need Berkshelf as dependency of Test-Kitchen 1.0. Do you want to propose some Test-Kitchen coverage for Travis VMs ?

@michaelklishin

@drnic with this repo being open source, people can contribute small improvements the Travis team has no time to handle (or is simply unaware of, like new version releases in Python and PHP communities). But the repo was not created to provide general purpose cookbooks. Nobody ever guaranteed this.

We extracted some cookbooks that seemed useful, "novel" or trivial to maintain. You can find them under github.com/michaelklishin. Everything else is either imported from upstream, heavily modified for Travis' needs or just not particularly interesting.

Lets accept one thing: the copy-paste-modify strategy is very common in the Chef community. So it's great that people can learn or look things up in this collection, and then modify to their needs. Whether it makes "software craftsmen" angry (not DRY, etc), it has worked fine so far for the Chef community in general and many individual Chef-provisioned projects I've been part of.

@joshk joshk reopened this
@joshk
Owner

I would still like to keep this open as some of the ideas in here are valid and interesting.

@hiremaga

re: @gildegoma

@mkocher & @pivotal's ciborg uses the cookbooks in this project to build a Jenkins based CI environment on EC2.

I attempted to do something similar with Vagrant. When I tried to use Librarian-Chef initially it wouldn't work due to the absence of metadata.rb files in some cookbooks. I was able to get this working by ensuring there's at least a minimal metadata.rb file in every cookbook. It seemed reasonable at the time to submit #170 since some cookbooks in this project already had them.

FWIW, Librarian handles multiple cookbooks in the same repo, e.g. https://github.com/hiremaga/ciborg-vagrant/blob/master/Cheffile

@gildegoma
Collaborator

Thanks everyone for your very informative inputs! (following is my own view, as usual)

Librarian handles multiple cookbooks in the same repo

Sure! It was a bit misleading to "occupy" this issue to open this discussion. Since topics are quite related, I took place here.

To summarize, we see there 3 variants to include (travis) cookbooks:

  1. git submodules: this is the "heavy" way, but it works. Example: https://github.com/pivotal/ciborg)
  2. basic bundling: Dependency configuration (Cheffile, Berksfile) must explicitly import all required cookbooks (no dependency inference, no version matching). Example: https://github.com/hiremaga/ciborg-vagrant
  3. advanced (or usual?) bundling: Dependency configuration relies on metadata like version, depends, conflicts, etc. to fetch the whole set of cookbooks. This is the "modern and powerful" way.

As @hiremaga argued, #170 is fully compatible with current style, and it easily makes travis-cookbooks "simple bundling" ready. The maintenance cost of this level is quite low, and thus should be supported now, I think. We could however argue that the benefits compared to "git submodules" are few (less stuff, faster download).

Since the Chef toolchain currently used in Travis provisioning context is not ensuring that all cookbooks are foodcritic-valid, I'm afraid that stepping now in changes like #176 would be like a bad promise, taking obvious risks to not honour the contract of "advanced bundling".

That said, I do think that Travis should (sooner or later) modernize and upgrade to tools like Berkshelf/Librarian (sweat integration/upgrades of external community cookbooks, instead self-maintenance of all cookbooks, even ones like 'apt' or 'build-essential') and Test Kitchen (having provisioning test suites for each kind of travis worker sounds great, especially for a continuous integration project!). @drnic: interested to work on a TestKitchen prototype for Travis? I am...

My concern was to better undertand the external usage kinds and to better figure out how Travis Project could manage to satisfy them, maintaining his own goals as top priority. I think we have some good overview now... Thanks again to everyone!

@gildegoma
Collaborator

In relation with these topics, I recently started to work on extending the travis-cookbooks toolchain (for cookbook quality assurance, not reusability purpose):

  1. Foodcritic lint tool has been added, but with all failing rules being disabled. Now, the idea is to go through this list and decide for each of these rules if it should be better:

    • fixed and checked (sooner or later), as shown in 83e0c07, or
    • ignored (on the long-term), as shown in ec68d4a

    I suggest that metadata.rb pull requests mentioned above should be considered as part of this Foodcritic process.

  2. in #186, I also started to experiment Test-Kitchen integration testing. Feedbacks are welcome! I had to make travis_build_environment attributes overridable, as @drnic did in #176, but I don't need to modify/add metadata.rb files because TK can be used without Berkshelf (or Librarian). Of course the complete run_list must be fully described. Feedbacks are welcome!

This was referenced
@patcon

Oh, I really need to refresh myself on this thread, but I'll have to do that later since I'm on vacation. For now, I just wanted to submit a possible strategy for slowly moving into usage of librarian-chef (or berkshelf):

Both tools can pull in local directories as cookbooks, so we could just port travis-cookbooks to use librarian-chef without actually using any external cookbooks to start. For example:

travis-cookbooks
├── .gitignore
├── Cheffile
├── Cheffile.lock
├── cookbooks
└── cookbooks-source
    ├── ant
    ├── apt
    ├── build-essential
    └── etc...
# Cheffile

#!/usr/bin/env ruby
#^syntax detection

site "http://community.opscode.com/api/v1"

cookbook "ant",
  :path => "cookbooks-source/ant"
cookbook "apt",
  :path => "cookbooks-source/apt"
cookbook "build-essential",
  :path => "cookbooks-source/build-essential"

# Etc...

We just move all the cookbooks from worker_host and ci_environment into cookbooks-source. If we want to be really conservation, we wouldn't even need to do that small migration first, and could leave them separate (altering librarian-chef's :path appropriately) until we've made sure that any depending project has been "retrofitted". (Also we'd need .gitignore to contain cookbooks and tmp so the librarian artifacts aren't version-controlled.)

If we did this, then we can easily start transitioning sensible cookbooks to start using their community counterparts. The benefit is that we can do it slowly an stepwise, at our own pace.

@patcon

For kicks, I just converted this. Currently, librarian-chef install won't run because there's a cyclic dep in php and composer, so that needs to be resolved first. But assuming that's the only issue, we can version-controll Cheffile.lock and be on our way. At this point, it shouldn't be a problem that cookbooks are defined twice in Cheffile (once for ci_environment and once for worker_host) because they should both be the same (I assume).

Now, external projects can start anticipating a single "cookbooks" dir having all cookbooks, and once we've removed all expectations of using the previous 2 subdirs, we can remove the doubles in the Cheffile. Then after that, we could think about moving them into something like a single cookbooks-source dir. Then finally, we could start migrating the simple cookbooks to use community ones.

Anyhow, as I said, I'm on vacation, but just wanted to toss out an option to painlessly migrate if we feel like it :)

@gildegoma gildegoma referenced this pull request from a commit
@roidrage roidrage Restore recommended dependency.
Breaks anything trying to use this cookbook without
having the apache2 cookbook around.
dbd3027
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 8, 2013
  1. @patcon
This page is out of date. Refresh to see the latest.
Showing with 127 additions and 1 deletion.
  1. +2 −1  .gitignore
  2. +124 −0 Cheffile
  3. +1 −0  Gemfile
View
3  .gitignore
@@ -8,4 +8,5 @@ tmp/*
/Gemfile.lock
/.rvmrc
/.bundle
-
+cookbooks/
+tmp/
View
124 Cheffile
@@ -0,0 +1,124 @@
+#!/usr/bin/env ruby
+#^syntax detection
+
+site 'http://community.opscode.com/api/v1'
+
+# cookbook 'chef-client'
+
+# cookbook 'apache2', '>= 1.0.0'
+
+# cookbook 'rvm',
+# :git => 'https://github.com/fnichol/chef-rvm'
+
+# cookbook 'postgresql',
+# :git => 'https://github.com/findsyou/cookbooks',
+# :ref => 'postgresql-improvements'
+cookbook "ant", :path => "ci_environment/ant"
+cookbook "apt", :path => "ci_environment/apt"
+cookbook "ark", :path => "ci_environment/ark"
+cookbook "bazaar", :path => "ci_environment/bazaar"
+cookbook "bison", :path => "ci_environment/bison"
+cookbook "build-essential", :path => "ci_environment/build-essential"
+cookbook "cassandra", :path => "ci_environment/cassandra"
+cookbook "chromium", :path => "ci_environment/chromium"
+cookbook "clang", :path => "ci_environment/clang"
+cookbook "cmake", :path => "ci_environment/cmake"
+cookbook "composer", :path => "ci_environment/composer"
+cookbook "couchdb", :path => "ci_environment/couchdb"
+cookbook "doxygen", :path => "ci_environment/doxygen"
+cookbook "elasticsearch", :path => "ci_environment/elasticsearch"
+cookbook "emacs", :path => "ci_environment/emacs"
+cookbook "erlang", :path => "ci_environment/erlang"
+cookbook "firefox", :path => "ci_environment/firefox"
+cookbook "git", :path => "ci_environment/git"
+cookbook "golang", :path => "ci_environment/golang"
+cookbook "gradle", :path => "ci_environment/gradle"
+cookbook "gvm", :path => "ci_environment/gvm"
+cookbook "haskell", :path => "ci_environment/haskell"
+cookbook "hbase", :path => "ci_environment/hbase"
+cookbook "htop", :path => "ci_environment/htop"
+cookbook "imagemagick", :path => "ci_environment/imagemagick"
+cookbook "iptables", :path => "ci_environment/iptables"
+cookbook "java", :path => "ci_environment/java"
+cookbook "jpackage", :path => "ci_environment/jpackage"
+cookbook "kerl", :path => "ci_environment/kerl"
+cookbook "kestrel", :path => "ci_environment/kestrel"
+cookbook "leiningen", :path => "ci_environment/leiningen"
+cookbook "libffi", :path => "ci_environment/libffi"
+cookbook "libgdbm", :path => "ci_environment/libgdbm"
+cookbook "libncurses", :path => "ci_environment/libncurses"
+cookbook "libossp-uuid", :path => "ci_environment/libossp-uuid"
+cookbook "libqt4", :path => "ci_environment/libqt4"
+cookbook "libreadline", :path => "ci_environment/libreadline"
+cookbook "libssl", :path => "ci_environment/libssl"
+cookbook "libxml", :path => "ci_environment/libxml"
+cookbook "libyaml", :path => "ci_environment/libyaml"
+cookbook "lighttpd", :path => "ci_environment/lighttpd"
+cookbook "maven3", :path => "ci_environment/maven3"
+cookbook "memcached", :path => "ci_environment/memcached"
+cookbook "mercurial", :path => "ci_environment/mercurial"
+cookbook "mingw32", :path => "ci_environment/mingw32"
+cookbook "mongodb", :path => "ci_environment/mongodb"
+cookbook "mysql", :path => "ci_environment/mysql"
+cookbook "neo4j-server", :path => "ci_environment/neo4j-server"
+cookbook "networking_basic", :path => "ci_environment/networking_basic"
+cookbook "nodejs", :path => "ci_environment/nodejs"
+cookbook "openssh", :path => "ci_environment/openssh"
+cookbook "openssl", :path => "ci_environment/openssl"
+cookbook "perlbrew", :path => "ci_environment/perlbrew"
+cookbook "phantomjs", :path => "ci_environment/phantomjs"
+cookbook "php", :path => "ci_environment/php"
+cookbook "phpbuild", :path => "ci_environment/phpbuild"
+cookbook "phpenv", :path => "ci_environment/phpenv"
+cookbook "postgresql", :path => "ci_environment/postgresql"
+cookbook "pypy", :path => "ci_environment/pypy"
+cookbook "python", :path => "ci_environment/python"
+cookbook "rabbitmq", :path => "ci_environment/rabbitmq"
+cookbook "ragel", :path => "ci_environment/ragel"
+cookbook "ramfs", :path => "ci_environment/ramfs"
+cookbook "rebar", :path => "ci_environment/rebar"
+cookbook "redis", :path => "ci_environment/redis"
+cookbook "riak", :path => "ci_environment/riak"
+cookbook "runit", :path => "ci_environment/runit"
+cookbook "rvm", :path => "ci_environment/rvm"
+cookbook "sbt", :path => "ci_environment/sbt"
+cookbook "scons", :path => "ci_environment/scons"
+cookbook "sphinx", :path => "ci_environment/sphinx"
+cookbook "sqlite", :path => "ci_environment/sqlite"
+cookbook "subversion", :path => "ci_environment/subversion"
+cookbook "sweeper", :path => "ci_environment/sweeper"
+cookbook "sysctl", :path => "ci_environment/sysctl"
+cookbook "timezone", :path => "ci_environment/timezone"
+cookbook "travis_build_environment", :path => "ci_environment/travis_build_environment"
+cookbook "unarchivers", :path => "ci_environment/unarchivers"
+cookbook "util-linux", :path => "ci_environment/util-linux"
+cookbook "vim", :path => "ci_environment/vim"
+cookbook "xserver", :path => "ci_environment/xserver"
+cookbook "zeromq", :path => "ci_environment/zeromq"
+cookbook "zookeeper", :path => "ci_environment/zookeeper"
+
+cookbook "ant", :path => "worker_host/ant"
+cookbook "apt", :path => "worker_host/apt"
+cookbook "build-essential", :path => "worker_host/build-essential"
+cookbook "chef-client", :path => "worker_host/chef-client"
+cookbook "chef_handler", :path => "worker_host/chef_handler"
+cookbook "collectd", :path => "worker_host/collectd"
+cookbook "collectd-librato", :path => "worker_host/collectd-librato"
+cookbook "disks", :path => "worker_host/disks"
+cookbook "emacs", :path => "worker_host/emacs"
+cookbook "git", :path => "worker_host/git"
+cookbook "iptables", :path => "worker_host/iptables"
+cookbook "java", :path => "worker_host/java"
+cookbook "jruby", :path => "worker_host/jruby"
+cookbook "monit", :path => "worker_host/monit"
+cookbook "networking_basic", :path => "worker_host/networking_basic"
+cookbook "ntp", :path => "worker_host/ntp"
+cookbook "runit", :path => "worker_host/runit"
+cookbook "rvm", :path => "worker_host/rvm"
+cookbook "ssh", :path => "worker_host/ssh"
+cookbook "sudo", :path => "worker_host/sudo"
+cookbook "travis_worker", :path => "worker_host/travis_worker"
+cookbook "travis_worker_collectd", :path => "worker_host/travis_worker_collectd"
+cookbook "users", :path => "worker_host/users"
+cookbook "vim", :path => "worker_host/vim"
+cookbook "virtualbox", :path => "worker_host/virtualbox"
View
1  Gemfile
@@ -1,6 +1,7 @@
source 'https://rubygems.org'
gem 'rake'
+gem "librarian-chef"
group :test do
gem 'foodcritic', '~> 2.1.0'
Something went wrong with that request. Please try again.