Infer more defaults (#582)

* Add default sources and protocol
* Fix yaml generation

README.md

@@ -18,20 +18,23 @@
 * [Why clowder](#why-clowder)
 * [Installation](#installation)
 * [The clowder.yml file](#the-clowderyml-file)
-  * [Syntax](#syntax)
 * [Command Usage](#command-usage)
   * [clowder init](#clowder-init)
   * [clowder herd](#clowder-herd)
   * [clowder status](#clowder-status)
   * [clowder forall](#clowder-forall)
   * [git commands](#git-commands)
   * [clowder repo commands](#clowder-repo-commands)
-  * [config commands](#config-commands)
+  * [config commands](#config-command
 * [Development](#development)
 
 ## Why clowder
 
-Managing multiple repositories can be pretty frustrating. There are a number of existing options, the primary being git submodules and subtrees. Google's [repo](https://code.google.com/p/git-repo) tool takes a different approach, but is closely tied to Google's development workflow. `clowder` uses a similar approach as `repo`, but without the ties to Google's Android workflows. Information about projects is specified in a `clowder.yml` file. Projects can track branches, or be tied to specific tags or commits. This file can be checked into its own repository so it can be versioned and shared across teams. The primary purpose of `clowder` is synchronization of multiple repositories, so normal development still takes place in individual repositories with the usual `git` commands.
+There are many ways to organize projects with git. Monorepos, submodules, subtrees, or [some](https://github.com/cristibalan/braid) [other](https://github.com/mixu/gr) [tool](https://github.com/ingydotnet/git-subrepo). `clowder` is one of these other tools. Its approach is heavily influeced by the [repo tool](https://gerrit.googlesource.com/git-repo) Google uses to manage the Android Open Source Project.
+
+Projects are specified in a `clowder.yml` file that can be checked into its own repo, allowing it to be shared across teams. `clowder` essentially makes this file executable, allowing commands to be run across projects. `clowder` can update submodules, lfs files, and custom git config entries. Projects can track branches, or be tied to specific tags or commits. Forks can be configured along with their upstream source, wherever they may live. Snapshots of project states can be saved for later restoration. And probably more things...
+
+Daily development still takes place in individual repos, with normal `git` commands. But `clowder` is there if you need to synchronize or run commands on multiple repos.
 
 ## Installation
 
@@ -54,89 +57,112 @@ sudo pip3 install clowder-repo --force-reinstall --pre
 
 ## The clowder.yml file
 
+For the full specification, see [the clowder.yml syntax reference](docs/clowder-yml-syntax-reference.md)
+
+An example `clowder.yml` for [some](https://github.com/llvm/llvm-project) [well](https://github.com/apple/swift)-[known](https://github.com/tensorflow/tensorflow) [projects](https://gerrit.googlesource.com/git-repo):
+
 ```yaml
-name: my-first-clowder
+name: cool-projects
 
 defaults:
+  branch: master
+  remote: origin
+  source: github
   protocol: ssh
 
 sources:
   - name: github
     url: github.com
+  - name: google
+    url: gerrit.googlesource.com
+    protocol: https
 
 projects:
-  - name: jrgoodle/mu
-  - name: jrgoodle/duke
-  - name: jrgoodle/kit
-    groups: [black-cats]
-  - name: jrgoodle/kishka
-    groups: [black-cats]
-  - name: jrgoodle/june
-    groups: [black-cats, notdefault]
-  - name: jrgoodle/sasha
-    groups: [black-cats, notdefault]
+  - name: llvm/llvm-project
+  - name: apple/swift
+  - name: tensorflow/tensorflow
+  - name: git-repo
+    path: repo
+    source: google
 ```
 
-### Syntax
-
-For more information, see [the clowder.yml syntax reference](docs/clowder-yml-syntax-reference.md)
+The `name` is simply a descriptive label. The `defaults` section contains the protocol to use for cloning repositories, the git remote and branch, and the source to clone from. `clowder` assumes the following defaults:
 
-```yaml
-name: my-first-clowder
-```
+* `remote`: `origin`
+* `branch`: `master`
+* `protocol`: `ssh`
+* `source`: `github`
 
-The `name` is simply a descriptive identifier. It must be a string that doesn't contain any spaces
+The `sources` section contains all the git hosting providers. The following sources are built in to `clowder`:
 
-```yaml
-defaults:
-  protocol: ssh
-```
+* `github`: `github.com`
+* `gitlab`: `gitlab.com`
+* `bitbucket`: `bitbucket.org`
 
-The `defaults` section requires the `protocol` to use for cloning repositories. The default `remote` is assumed to be `origin` and the default `branch` is assumed to to be `master`
+So the previous `clowder.yml` can be simplified to:
 
 ```yaml
+name: cool-projects
+
 sources:
-  - name: github
-    url: github.com
+  - name: google
+    url: gerrit.googlesource.com
+    protocol: https
+
+projects:
+  - name: llvm/llvm-project
+  - name: apple/swift
+  - name: tensorflow/tensorflow
+  - name: git-repo
+    path: repo
+    source: google
 ```
 
-The `sources` section contains all the locations to clone repositories from.
+A project requires a `name`, the path component of the git clone url. This is combined with `defaults.protocol` or `sources.protocol` to form the full git clone url, taking the form of  `git@${sources.url}:${projects.name}.git` or `https://${sources.url}/${projects.name}.git`. If `path` is not specified, the last component of the name is used for the local directory.
+
+In order to be able to run commands for only certain sets of projects, there are groups:
 
 ```yaml
+name: cool-projects
+
+sources:
+  - name: google
+    url: gerrit.googlesource.com
+    protocol: https
+
 projects:
-  - name: jrgoodle/mu
-  - name: jrgoodle/duke
-  - name: jrgoodle/kit
-    groups: [black-cats]
-  - name: jrgoodle/kishka
-    groups: [black-cats]
-  - name: jrgoodle/june
-    groups: [black-cats, notdefault]
-  - name: jrgoodle/sasha
-    groups: [black-cats, notdefault]
+  - name: llvm/llvm-project
+    groups: [notdefault, clattner]
+  - name: apple/swift
+    groups: [clattner]
+  - name: tensorflow/tensorflow
+    groups: [google, clattner]
+  - name: git-repo
+    path: repo
+    source: google
+    groups: [google]
 ```
 
-A project requires at minimum the `name` of the repository. This is combined with `defaults.protocol` or `sources.protocol` to form the full url for cloning the repository, taking the form of  `git@${sources.url}:${projects.name}.git` or `https://${sources.url}/${projects.name}.git`, depending on the protocol specified. If no `path` is specified, the last component of the `name` is used for the directory the project is cloned to. Projects can specify custom `groups` in order to run commands for only certain projects. By default, all projects are added to the `all` group, and a group of their `name` and `path`. If `notdefault` is present, then the project will not be included in commands unless another `group` argument is given that it belongs to.
-
-There's much more cusomization possible with `clowder`. For some more complex examples see:
+Projects are automatically added to the `all` group, a group of their `name`, and a group of their `path`. If `notdefault` is specified, the project will not be included in commands unless another group argument is given that it belongs to.
 
-[Cats clowder.yml example](docs/clowder-yml-cats.md)
+For some more custom examples, see:
 
-[Forks clowder.yml example](docs/clowder-yml-forks.md)
+* [Cats clowder.yml example](docs/clowder-yml-cats.md)
+* [Forks clowder.yml example](docs/clowder-yml-forks.md)
 
 ## Command Usage
 
-For more information, see [the commands doc](docs/commands.md)
+For the full command reference, see [the commands doc](docs/commands.md)
 
-First create a directory where all the projects will be cloned:
+The following examples use an [existing repo](https://github.com/JrGoodle/clowder-examples) containing a [clowder.yml](https://github.com/JrGoodle/clowder-examples/blob/master/clowder.yml) file.
+
+First, create a directory where all the projects will be cloned:
 
 ```bash
 mkdir cats
 cd cats
 ```
 
-It's possible to just create a local `clowder.yml` file, but it's recommended to check the file into the root of a dedicated repository. These examples use an [existing repository](https://github.com/JrGoodle/clowder-examples) containing a [clowder.yml](https://github.com/JrGoodle/clowder-examples/blob/master/clowder.yml) file.
-
 ### clowder init
 
 ```bash
@@ -145,7 +171,7 @@ clowder init git@github.com:JrGoodle/clowder-examples.git
 
 The `clowder init` command does the following:
 
-* Clones the [examples clowder repo](https://github.com/JrGoodle/clowder-examples) in the `cats/.clowder` directory
+* Clones the [examples clowder repo](https://github.com/JrGoodle/clowder-examples) in the `.clowder` directory
 * Creates a symlink in the `cats` directory: `clowder.yml` -> `.clowder/clowder.yml`
 
 ![clowder init](docs/examples/clowder-init.gif)

clowder_test/clowder_test/cli/misc_controller.py

@@ -34,6 +34,14 @@ def all(self) -> None:
 
         self._execute_command('./test_example_misc.sh', ROOT_DIR / 'test' / 'scripts')
 
+    @expose(
+        help='Run misc defaults tests'
+    )
+    def defaults(self) -> None:
+        """clowder misc defaults tests"""
+
+        self._execute_command('./defaults.sh', self.path)
+
     @expose(
         help='Run misc forks tests'
     )

docs/clowder-yml-cats.md

@@ -6,16 +6,12 @@ See also: [clowder.yml syntax reference](clowder-yml-syntax-reference.md)
 name: cats-clowder
 
 defaults:
-  protocol: ssh
+  protocol: https
   branch: tracking_branch
   git:
     config:
       alias.cat: '!echo "😸"'
 
-sources:
-  - name: github
-    url: github.com
-
 projects:
   - name: jrgoodle/mu
     branch: groom
@@ -46,14 +42,14 @@ projects:
 
 ```yaml
 defaults:
-  protocol: ssh
+  protocol: https
   branch: tracking_branch
   git:
     config:
       alias.cat: '!echo "😸"'
 ```
 
-This example specifies a default `branch` of `tracking_branch` that all projects will inherit. It's also possible to specify a default `tag` or `commit`. The `git`section can contain custom git config entries that will be installed for all projects.
+This example specifies a default branch of `tracking_branch` that all projects will inherit. It's also possible to specify a default `tag` or `commit`. The `git`section can contain custom git config entries that will be installed for all projects.
 
 ```yaml
 - name: jrgoodle/mu
@@ -78,7 +74,7 @@ This project will check out the repository to the commit the `v0.01` tag points
   commit: f2e20031ddce5cb097105f4d8ccbc77f4ac20709
 ```
 
-This project will check out the repository to the commit specified by the full sha-1. The `path` the repository will be cloned at is `black-cats/kit`
+This project will check out the repository to the commit specified by the full sha-1. The path the repository will be cloned at is `black-cats/kit`
 
 ```yaml
 - name: jrgoodle/kishka

docs/clowder-yml-forks.md

@@ -6,14 +6,10 @@ See also: [clowder.yml syntax reference](clowder-yml-syntax-reference.md)
 name: forks-clowder
 
 defaults:
-  protocol: ssh
-  source: github
   git:
     recursive: true
 
 sources:
-  - name: github
-    url: github.com
   - name: chromium
     url: chromium.googlesource.com
     protocol: https
@@ -44,18 +40,14 @@ projects:
 
 ```yaml
 defaults:
-  protocol: ssh
-  source: github
   git:
     recursive: true
 ```
 
-Because more than one source is specified, the default `source` to use to clone projects must be given. All projects will recursively init and update submodules by default.
+All projects will recursively init and update submodules by default.
 
 ```yaml
 sources:
-  - name: github
-    url: github.com
   - name: chromium
     url: chromium.googlesource.com
     protocol: https
@@ -73,7 +65,7 @@ The `protocol` is set specifically to `https` for repositories where the user ma
     name: JrGoodle/djinni
 ```
 
-The original repository will be cloned with a remote named `upstream`. The `fork` inherits the default remote `origin`.
+The original repository will be cloned with a remote named `upstream`. The `fork` uses the default remote `origin`.
 
 ```yaml
 - name: external/gyp
@@ -95,4 +87,4 @@ The user's fork will track `fork-branch`.
     name: JrGoodle/sox
 ```
 
-This example will check out the repository to the `sox` directory, rather than `code`.
+The project is cloned to to the `sox` directory, rather than `code`.