Skip to content

Commit

Permalink
Update README
Browse files Browse the repository at this point in the history
  • Loading branch information
@JrGoodle
JrGoodle committed Sep 22, 2020
1 parent ee94492 commit 5914ed3
Show file tree
Hide file tree
Showing 3 changed files with 201 additions and 58 deletions.
20 changes: 17 additions & 3 deletions .codeclimate.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@ checks:
# config:
# threshold: 20

# exclude_patterns:
# - "tests/"
# - "script/"
exclude_patterns:
- "tests/"
- "script/"
- "docs/"
- ".*"
- "**/*.json"
- "**/*.yml"
- "**/*.ini"
- "**/*.txt"
- "**/*.cfg"
- "**/*.yaml"
- "Brew*"
- "Docker*"
- "CHANGELOG.md"
- "CONTRIBUTING.md"
- "LICENSE"
- "README.md"
4 changes: 4 additions & 0 deletions .markdownlint.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"no-inline-html": false,
"line-length": false
}
235 changes: 180 additions & 55 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,20 @@
# `clowder`

[![GitHub Actions Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2FJrGoodle%2Fclowder%2Fbadge&style=flat)](https://actions-badge.atrox.dev/JrGoodle/clowder/goto)
[![CircleCI](https://circleci.com/gh/JrGoodle/clowder.svg?style=shield)](https://circleci.com/gh/JrGoodle/clowder)
[![Code Climate Maintainability](https://api.codeclimate.com/v1/badges/56c92799de08f9ef9258/maintainability)](https://codeclimate.com/github/JrGoodle/clowder/maintainability)
[![codecov coverage](https://codecov.io/gh/JrGoodle/clowder/branch/master/graph/badge.svg)](https://codecov.io/gh/JrGoodle/clowder)
[![PyPI version](https://badge.fury.io/py/clowder-repo.svg)](https://badge.fury.io/py/clowder-repo)
[![Python versions](https://img.shields.io/pypi/pyversions/clowder-repo.svg)](https://pypi.python.org/pypi/clowder-repo)
[![Requirements Status](https://requires.io/github/JrGoodle/clowder/requirements.svg?branch=master)](https://requires.io/github/JrGoodle/clowder/requirements/?branch=master)
[![Documentation Status](https://readthedocs.org/projects/clowder/badge/?version=latest)](http://clowder.readthedocs.io)

> **clowder** - A group of cats
>
> **herding cats** - An idiom that refers to a frustrating attempt to control or organize a class of entities which are uncontrollable or chaotic
| | |
|:-|:-|
| docs | [![Documentation Status](https://readthedocs.org/projects/clowder/badge/?version=latest)](http://clowder.readthedocs.io) |
| tests | [![GitHub Actions Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2FJrGoodle%2Fclowder%2Fbadge&style=flat)](https://actions-badge.atrox.dev/JrGoodle/clowder/goto) [![CircleCI](https://circleci.com/gh/JrGoodle/clowder.svg?style=shield)](https://circleci.com/gh/JrGoodle/clowder) [![Code Climate Maintainability](https://api.codeclimate.com/v1/badges/56c92799de08f9ef9258/maintainability)](https://codeclimate.com/github/JrGoodle/clowder/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/56c92799de08f9ef9258/test_coverage)](https://codeclimate.com/github/JrGoodle/clowder/test_coverage) |
| package | [![PyPI version](https://badge.fury.io/py/clowder-repo.svg)](https://badge.fury.io/py/clowder-repo) [![Python versions](https://img.shields.io/pypi/pyversions/clowder-repo.svg)](https://pypi.python.org/pypi/clowder-repo) [![License](https://img.shields.io/pypi/l/clowder-repo.svg)](https://pypi.python.org/pypi/clowder-repo) [![Status](https://img.shields.io/pypi/status/clowder-repo.svg)](https://pypi.python.org/pypi/clowder-repo) [![Requirements Status](https://requires.io/github/JrGoodle/clowder/requirements.svg?branch=master)](https://requires.io/github/JrGoodle/clowder/requirements/?branch=master) |

## Table of Contents

* [Why clowder](#why-clowder)
* [Installation](#installation)
* [The clowder.yml file](#the-clowderyml-file)
* [clowder.yml](#clowderyml)
* [Command Usage](#command-usage)
* [clowder init](#clowder-init)
* [clowder herd](#clowder-herd)
Expand All @@ -32,7 +29,7 @@

There are many ways to organize projects with git. Monorepos, submodules, subtrees, or [some](https://github.com/cristibalan/braid) [other](https://github.com/mixu/gr) [tool](https://github.com/ingydotnet/git-subrepo). `clowder` is one of the other tools. Its approach is heavily influeced by the [repo tool](https://gerrit.googlesource.com/git-repo) Google uses to manage the Android Open Source Project.

Projects are listed in a `clowder.yml` file that can be checked into its own repo, allowing it to be shared across teams. `clowder` essentially makes this file executable, allowing commands to be run across projects. `clowder` can update submodules, lfs files, and custom git config entries. Projects can track branches, or be tied to specific tags or commits. Upstreams can be configured along with their upstream source, wherever they may live. Snapshots of project states can be saved for later restoration. And more...
Projects information is stored in a `clowder.yml` file. If checked into its own repo, it can be shared among users. `clowder` essentially makes this file executable, allowing commands to be run across projects. `clowder` can update submodules, lfs files, and custom git config entries. Projects can track branches, or be tied to specific tags or commits. Upstreams can be configured along with their upstream source, wherever they may live. Snapshots of project states can be saved for later restoration. And more...

Daily development still takes place in individual repos, with normal `git` commands. But `clowder` is there if you need to synchronize or run commands on multiple repos.

Expand All @@ -55,7 +52,13 @@ To install the latest pre-release version:
sudo pip3 install clowder-repo --force-reinstall --pre
```

## The clowder.yml file
For command autocompletion, add to your shell profile:

```bash
command -v clowder >/dev/null 2>&1 && eval "$(register-python-argcomplete clowder)"
```

## clowder.yml

For more inforrmation:

Expand All @@ -73,11 +76,15 @@ clowder:
- tensorflow/tensorflow
```

Although terse, this is enough to enable `clowder`. With the ommitted default settings:
The name is simply a descriptive label. Projects are specified by the last components of the git clone url.

If the ommitted default settings are included:

```yaml
name: cool-projects

protocol: ssh

sources:
github:
url: github.com
Expand All @@ -88,25 +95,34 @@ defaults:
branch: master

clowder:
- llvm/llvm-project
- apple/swift
- tensorflow/tensorflow
- name: llvm/llvm-project
path: llvm-project
- name: apple/swift
path: swift
- name: tensorflow/tensorflow
path: tensorflow
```

The `name` is simply a descriptive label. The `defaults` section contains the git branch and remote, the source to clone from, and the protocol to use for cloning repositories. `clowder` assumes the following defaults:
The defaults section contains the git branch and remote, the source to clone from, and the protocol to use for cloning repositories.

A project requires at minimum a name (the last components of the git clone url). If the project path is not specified, the last component of the project name is used for the local directory.

* `branch`: `master`
* `remote`: `origin`
* `source`: `github`
* `protocol`: `ssh`
The sources section is where custom git hosting providers are specified. The following sources are built in:

The `sources` section contains all the git hosting providers. The following sources are built in to `clowder`:
| `source.name` | `source.url` |
| ------------- | --------------- |
| `github` | `github.com` |
| `gitlab` | `gitlab.com` |
| `bitbucket` | `bitbucket.org` |

* `github`: `github.com`
* `gitlab`: `gitlab.com`
* `bitbucket`: `bitbucket.org`
Depending on the protocol, the project name is combined with the source url to form the full git clone url:

If we wanted to add a project at another hosting site:
| protocol | git url |
| -------- | ------------------------------------------- |
| ssh | `git@${source.url}:${project.name}.git` |
| https | `https://${source.url}/${project.name}.git` |

To add a project from a custom hosting location:

```yaml
name: cool-projects
Expand All @@ -125,52 +141,137 @@ clowder:
source: google
```

A project requires a `name`, the path component of the git clone url. This is combined with `defaults.protocol` or `sources.protocol` to form the full git clone url, taking the form of `git@${sources.url}:${projects.name}.git` or `https://${sources.url}/${projects.name}.git`. If `path` is not specified, the last component of the name is used for the local directory.
or equivalently:

```yaml
name: cool-projects

clowder:
- llvm/llvm-project
- apple/swift
- tensorflow/tensorflow
- name: git-repo
path: repo
source:
url: gerrit.googlesource.com
protocol: https
```

In order to be able to run commands for only certain sets of projects, there are groups:

```yaml
name: cool-projects

sources:
google:
url: gerrit.googlesource.com
protocol: https

clowder:
- name: llvm/llvm-project
groups: [notdefault, clattner]
groups: [clattner]
- name: apple/swift
groups: [clattner]
- name: tensorflow/tensorflow
groups: [google, clattner]
- name: git-repo
path: repo
source: google
groups: [google]
groups: [google, notdefault]
source:
url: gerrit.googlesource.com
protocol: https
```

or equivalently with grouped sections:

```yaml
name: cool-projects

clowder:
clattner:
- name: llvm/llvm-project
groups: [notdefault]
- llvm/llvm-project
- apple/swift
- name: tensorflow/tensorflow
groups: [google]
google:
- name: git-repo
path: repo
groups: [notdefault]
source:
url: gerrit.googlesource.com
protocol: https
```

Projects are automatically added to the `all` group, a group of their `name`, and a group of their `path`. If `notdefault` is specified, the project will not be included in commands unless another group argument is given that it belongs to.
or equivalently using grouped section settings:

For some more custom examples, see:
```yaml
name: cool-projects

sources:
google:
url: gerrit.googlesource.com
protocol: https

clowder:
clattner:
- llvm/llvm-project
- apple/swift
- name: tensorflow/tensorflow
groups: [google]
google:
defaults:
source: google
groups: [notdefault]
projects:
- name: git-repo
path: repo
```

Projects are automatically added to the `all` group, and to groups of their name and path. If `notdefault` is specified, the project will not be included in commands unless it belongs to another supplied group argument.

To save projects to a subdirectory:

```yaml
name: cool-projects

clowder:
clattner:
path: clattner
projects:
- llvm/llvm-project
- apple/swift
- name: tensorflow/tensorflow
groups: [google]
google:
path: google
projects:
- name: git-repo
path: repo
groups: [notdefault]
source:
url: gerrit.googlesource.com
protocol: https
```

which is equivalent to:

```yaml
name: cool-projects

clowder:
clattner:
- name: llvm/llvm-project
path: clattner/llvm-project
- name: apple/swift
path: clattner/swift
- name: tensorflow/tensorflow
path: clattner/tensorflow
groups: [google]
google:
- name: git-repo
path: google/repo
groups: [notdefault]
source:
url: gerrit.googlesource.com
protocol: https
```

For more examples, see:

* [Cats clowder.yml example](docs/clowder-yml-cats.md)
* [Upstreams clowder.yml example](docs/clowder-yml-upstreams.md)
Expand All @@ -181,7 +282,7 @@ For the full command reference, see [the commands doc](docs/commands.md)

The following examples use an [existing repo](https://github.com/JrGoodle/clowder-examples) containing a [clowder.yml](https://github.com/JrGoodle/clowder-examples/blob/master/clowder.yml) file.

First, create a directory where all the projects will be cloned:
To follow along, create a directory where all the projects will be cloned:

```bash
mkdir cats
Expand Down Expand Up @@ -222,31 +323,50 @@ The `clowder status` command prints the current state of all projects.
### clowder forall

```bash
clowder forall -c 'git status' # Run command in all project directories
# Run command in all project directories
clowder forall -c 'git status'
```

### git commands

For more information, see [the commands doc](docs/commands.md#git-commands)

```bash
clowder branch # Print all local branches
clowder checkout 'my_branch' # Checkout 'my_branch' in projects if it exists
clowder clean # Discard any changes in projects
clowder diff # Print git diff for all projects
clowder start 'my_feature' # Create new branch 'my_feature' for all projects
clowder stash # Stash changes in all projects
clowder prune 'stale_branch' # Prune branch 'stale_branch' for all projects
# Print all local branches
clowder branch

# Checkout 'my_branch' in projects if it exists
clowder checkout 'my_branch'

# Discard any changes in projects
clowder clean

# Print git diff for all projects
clowder diff

# Create new branch 'my_feature' for all projects
clowder start 'my_feature'

# Stash changes in all projects
clowder stash

# Prune branch 'stale_branch' for all projects
clowder prune 'stale_branch'
```

### clowder repo commands

For more information, see [the commands doc](docs/commands.md#clowder-repo-commands)

```bash
clowder link 'v0.1' # Set clowder.yml symlink to a previously saved version
clowder repo run 'git status' # Run command in .clowder directory
clowder save 'v0.1' # Save a version of clowder.yml with current commit sha's
# Set clowder.yml symlink to a previously saved version
clowder link 'v0.1'

# Run command in .clowder directory
clowder repo run 'git status'

# Save a version of clowder.yml with current commit sha's
clowder save 'v0.1'
```

### config commands
Expand All @@ -256,11 +376,16 @@ clowder save 'v0.1' # Save a version of clowder.yml with current commit sha's
For more information, see [the commands doc](docs/commands.md#clowder-config)

```bash
clowder config get # Get config values
clowder config set projects 'my_group' # Set config values
clowder config clear projects # Clear config values
# Get config values
clowder config get

# Set config values
clowder config set projects 'my_group'

# Clear config values
clowder config clear projects
```

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) for information on setting up your environment for development and contribution guidelines
See [CONTRIBUTING.md](CONTRIBUTING.md)

0 comments on commit 5914ed3

Please sign in to comment.