Skip to content

Commit

Permalink
Apply suggestions from code Review
Browse files Browse the repository at this point in the history
- Change some docs structure
- Fix some broken links
- Add a brief quickstart in the quickstart
- Fix Title Capitalization

Signed-off-by: Carlos Martín <carlos.martin.sanchez@gmail.com>
Signed-off-by: David Pordomingo <David.Pordomingo.F@gmail.com>
  • Loading branch information
dpordomingo committed Jun 25, 2019
1 parent 01902be commit ee704a9
Show file tree
Hide file tree
Showing 15 changed files with 79 additions and 75 deletions.
32 changes: 17 additions & 15 deletions README.md
Expand Up @@ -21,17 +21,15 @@

## Introduction

**source{d} Community Edition (CE)** helps you to manage all your code and engineering data in one place powerful:
**source{d} Community Edition (CE)** helps you to manage all your code and engineering data in one place:

- **Code Retrieval**: Retrieve and store the git history of the code of your organization as a dataset.
- **Analysis in/for any Language**: Automatically identify languages, parse source code, and extract the pieces that matter in a distributed and language-agnostic way.
- **History Analysis**: Extract information from the evolution, commits, and metadata of your codebase and from Github, generating detailed reports and insights.
- **Analysis in/for any Language**: Automatically identify languages, parse source code, and extract the pieces that matter in a language-agnostic way.
- **History Analysis**: Extract information from the evolution, commits, and metadata of your codebase and from GitHub, generating detailed reports and insights.
- **Familiar APIs**: Analyze your code through powerful SQL queries. Use tools you're familiar with to create reports and dashboards.

This repository contains the code of **source{d} Community Edition (CE)** and its project documentation, which you can also see properly rendered at [docs.sourced.tech/community-edition](https://docs.sourced.tech/community-edition).

Currently, **source{d} CE** is in development process.


### Contents

Expand All @@ -47,17 +45,22 @@ Currently, **source{d} CE** is in development process.

**source{d} CE** supports Linux, macOS, and Windows.

You will find in the [Quick Start Guide](docs/quickstart/README.md) all the steps to get started with **source{d} CE**, from the installation of its dependencies to running SQL queries to inspect git repositories.
To run it you only need:

Once you are able to run the UI, you may find useful the following sections:
1. To have Docker installed in your PC
1. Download `sourced` binary (for your OS) from [our releases](https://github.com/src-d/sourced-ce/releases)
1. Run it:
```bash
$ sourced orgs init --token=<github_token> <github_org_name>
```
And log in into http://localhost:8088 with login: `admin`, and password: `admin`.

* [First steps](docs/usage/README.md)
* [Advanced usage](docs/advanced/README.md)
If you want more details of each step, you will find in the [**Quick Start Guide**](docs/quickstart/README.md) all the steps to get started with **source{d} CE**, from the installation of its dependencies to running SQL queries to inspect git repositories.


## Architecture

_For more details on the architecture of this project, read [docs/architecture.md](docs/advanced/architecture.md)._
_For more details on the architecture of this project, read [docs/architecture.md](docs/architecture.md)._

**source{d} CE** is deployed as Docker containers, using Docker Compose.

Expand All @@ -67,11 +70,10 @@ The main entry point of **source{d} CE** is [sourced-ui](https://github.com/src-

The data exposed by the web interface is prepared and processed by the following services:

- [babelfish](https://doc.bblf.sh): universal code parser,
- [babelfish-web](https://github.com/bblfsh/web): web client for Babelfish server,
- [gitcollector](https://github.com/src-d/gitcollector): fetches the git repositories owned by your organization,
- [ghsync](https://github.com/src-d/ghsync): fetches metadata from GitHub (users, pull requests, issues...)
- [gitbase](https://github.com/src-d/gitbase): SQL database interface to Git repositories
- [babelfish](https://doc.bblf.sh): universal code parser.
- [gitcollector](https://github.com/src-d/gitcollector): fetches the git repositories owned by your organization.
- [ghsync](https://github.com/src-d/ghsync): fetches metadata from GitHub (users, pull requests, issues...).
- [gitbase](https://github.com/src-d/gitbase): SQL database interface to Git repositories.


## Contributing
Expand Down
3 changes: 1 addition & 2 deletions docs/CONTRIBUTING.md
Expand Up @@ -6,8 +6,7 @@ As all source{d} projects, this project follows the

# Additional Contribution Guidelines

In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md),
this project follows the following guidelines.
In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md), this project follows the following guidelines.


## Changelog
Expand Down
16 changes: 8 additions & 8 deletions docs/README.md
Expand Up @@ -2,17 +2,17 @@
- [Dependencies](./quickstart/1-install-requirements.md)
- [Install **source{d} CE**](./quickstart/2-install-sourced.md)
- [Run **source{d} CE**](./quickstart/3-init-sourced.md)
- [Explore your data](./quickstart/4-explore-sourced.md)
* [First steps](./usage/README.md)
- [Multiple datasets](./usage/multiple-datasets.md)
- [SQL examples](./usage/examples.md)
- [sourced sub-commands](./usage/commands.md)
* [Advanced usage](./advanced/README.md)
- [Architecture](./advanced/architecture.md)
- [Babelfish UAST](./advanced/bblfsh.md)
- [Explore Your Data](./quickstart/4-explore-sourced.md)

## Advanced Usage
* [sourced Command Reference](./usage/commands.md)
* [Multiple Datasets](./usage/multiple-datasets.md)
* [SQL Examples](./usage/examples.md)
* [Babelfish UAST](./usage/bblfsh.md)

## Learn More

* [Architecture](./architecture.md)
* [Contribute](./CONTRIBUTING.md)
* [License](../LICENSE.md)

Expand Down
6 changes: 0 additions & 6 deletions docs/advanced/README.md

This file was deleted.

14 changes: 7 additions & 7 deletions docs/advanced/architecture.md → docs/architecture.md
Expand Up @@ -11,7 +11,7 @@ is the user's main interaction mechanism with **source{d} CE**.
It is also the only piece (other than Docker) that the user will need to explicitly
download on their machine to get started.

The `sourced` binary manages the different installed environments and its
The `sourced` binary manages the different installed environments and their
configurations, acting as a wrapper of Docker Compose.

The whole architecture is based on Docker containers, orchestrated by Docker Compose
Expand All @@ -20,9 +20,9 @@ and managed by `sourced`.

## Docker Set Up

In order to make this work in the easiest way, there was made some design decisions:
In order to make this work in the easiest way, some design decisions were made:

### Isolated environments.
### Isolated Environments.

_Read more in [Working With Multiple Data Sets](../usage/multiple-datasets.md)_

Expand Down Expand Up @@ -54,12 +54,12 @@ for this purpose:

## Persistence

To prevent losing data when restarting services, or upgrading containers, its data
To prevent losing data when restarting services, or upgrading containers, the data
is stored in volumes. These volumes also share the same prefix with the containers
in the same environment, e.g. `srcd-c3jjlwq_gitbase_repositories`.

These are the most relevant volumes:
- `gitbase_repositories`, stores the repositories to be analyzed,
- `gitbase_indexes`, stores the gitbases indexes,
- `gitbase_repositories`, stores the repositories to be analyzed
- `gitbase_indexes`, stores the gitbases indexes
- `metadata`, stores the metadata from GitHub pull requests, issues, users...
- `postgres`, stores the dashboards and charts used by the web interface.
- `postgres`, stores the dashboards and charts used by the web interface
10 changes: 5 additions & 5 deletions docs/quickstart/1-install-requirements.md
@@ -1,8 +1,8 @@
# Install source{d} Community Edition dependencies
# Install source{d} Community Edition Dependencies

## Install Docker

_Please note that Docker Toolbox is supported neither for Windows nor for macOS. In case that you're running Docker Toolbox, please consider updating to newer Docker Desktop for Mac or Docker Desktop for Windows._
_Please note that Docker Toolbox is not supported. In case that you're running Docker Toolbox, please consider updating to newer Docker Desktop for Mac or Docker Desktop for Windows._

Follow the instructions based on your OS:

Expand All @@ -11,8 +11,8 @@ Follow the instructions based on your OS:
- [Docker for macOS](https://store.docker.com/editions/community/docker-ce-desktop-mac)
- [Docker Desktop for Windows](https://hub.docker.com/editions/community/docker-ce-desktop-windows) (Make sure to read the [system requirements for Docker on Windows](https://docs.docker.com/docker-for-windows/install/).)

## Docker compose
## Docker Compose

**source{d} CE** is deployed as Docker containers, using [Docker Compose](https://docs.docker.com/compose), but it is not required a local installation of Docker Compose; if it is not found it will be deployed inside a container.
**source{d} CE** is deployed as Docker containers, using [Docker Compose](https://docs.docker.com/compose), but it is not required to have a local installation of Docker Compose; if it is not found it will be deployed inside a container.

If you prefer a local installation of Docker Compose, you can follow [Docker Compose install guide](https://docs.docker.com/compose/install)
If you prefer a local installation of Docker Compose, you can follow the [Docker Compose install guide](https://docs.docker.com/compose/install)
4 changes: 2 additions & 2 deletions docs/quickstart/2-install-sourced.md
Expand Up @@ -2,7 +2,7 @@

Download the **[latest release](https://github.com/src-d/sourced-ce/releases/latest)** for your Linux, macOS (Darwin) or Windows.

## on Linux or macOS
## On Linux or macOS

Extract `sourced` binary from the release you downloaded, and move it into your bin folder to make it executable from any directory:

Expand All @@ -11,7 +11,7 @@ $ tar -xvf path/to/sourced-ce_REPLACE-VERSION_REPLACE-OS_amd64.tar.gz
$ sudo mv path/to/sourced-ce_REPLACE-OS_amd64/sourced /usr/local/bin/
```

## on Windows
## On Windows

*Please note that from now on we assume that the commands are executed in `powershell` and not in `cmd`.*

Expand Down
18 changes: 13 additions & 5 deletions docs/quickstart/3-init-sourced.md
@@ -1,7 +1,7 @@
# Initialize source{d} Community Edition

_For the full list of the sub-commands offered by `sourced`, please take a look
into [the `sourced` sub-commands inventory](../usage/commands.md)._
at [the `sourced` sub-commands inventory](../usage/commands.md)._

**source{d} CE** can be initialized from 2 different data sources: GitHub organizations, or local Git repositories.

Expand All @@ -10,10 +10,10 @@ Please note that you have to choose one data source to initialize **source{d} CE
**source{d} CE** will download and install Docker images on demand. Therefore, the first time you run some of these commands, they might take a bit of time to start up. Subsequent runs will be faster.


#### From GitHub Organizations
## From GitHub Organizations

When using GitHub organizations to populate the **source{d} CE** database you only need to provide a list of organization names and a [GitHub personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/). If no scope is granted to the user token, only public
data will be fetched. To let **source{d} CE** to access to also private repos and hidden users, the token should
data will be fetched. To let **source{d} CE** access also private repositories and hidden users, the token should
have the following scopes enabled:

- `repo` Full control of private repositories
Expand All @@ -26,10 +26,10 @@ Use this command to initialize, e.g.
$ sourced orgs init --token <token> src-d,bblfsh
```

It will also download, on background, the repositories of the passed GitHub organizations, and its metadata: pull requests, issues, users...
It will also download, in the background, the repositories of the passed GitHub organizations, and their metadata: pull requests, issues, users...


#### From Local Repositories
## From Local Repositories

```shell
$ sourced init </path/to/repositories>
Expand All @@ -41,3 +41,11 @@ It will initialize **source{d} CE** to analyze the git repositories under the pa
Docker for Mac [requires enabling file sharing](https://docs.docker.com/docker-for-mac/troubleshoot/#volume-mounting-requires-file-sharing-for-any-project-directories-outside-of-users) for any path outside of `/Users`.

**Note for Windows:** Docker for Windows [requires shared drives](https://docs.docker.com/docker-for-windows/#shared-drives). Other than that, it's important to use a working directory that doesn't include any sub-directory whose access is not readable by the user running `sourced`. For example, using `C:\Users` as workdir, will most probably not work. For more details see [this issue](https://github.com/src-d/engine/issues/250).


## What's Next?

Once **source{d} CE** has been initialized, it will automatically open the web UI.
If the UI is not opened automatically, you can use `sourced web` command, or visit http://localhost:8088.

Use login: `admin` and password: `admin`, to access the web interface.
12 changes: 6 additions & 6 deletions docs/quickstart/4-explore-sourced.md
@@ -1,14 +1,14 @@
# Explore source{d} CE Web Interface

Once **source{d} CE** has been [installed with `sourced init`](./3-init-sourced.md), it will automatically open the web UI. If the UI wasn't opened automatically, you can use `sourced web` command, or visit http://localhost:8088.
Once **source{d} CE** has been [initialized with `sourced init`](./3-init-sourced.md), it will automatically open the web UI. If the UI is not automatically opened, you can use `sourced web` command, or visit http://localhost:8088.

Use login: `admin` and password: `admin`, to access the web interface.

The most relevant features that **source{d} CE** Web Interface offers are:
- **[SQL Lab](#sql-lab-querying-code-and-metadata)**, to query your repositories and its GitHub metadata,
- **[Babelfish web](#uast-parsing-code)**, web interface to parse files into UAST,
- **[Dashboards](#dashboards)**, to aggregate charts for exploring and visualizing your data,
- **Charts**, to see your data with a rich set of data visualizations,
- **[SQL Lab](#sql-lab-querying-code-and-metadata)**, to query your repositories and its GitHub metadata.
- **[Babelfish web](#uast-parsing-code)**, web interface to parse files into UAST.
- **[Dashboards](#dashboards)**, to aggregate charts for exploring and visualizing your data.
- **Charts**, to see your data with a rich set of data visualizations.
- A flexible UI to manage users, data sources, export data...

The user interface is based in the open-sourced [Apache Superset](http://superset.incubator.apache.org), so you can also refer to their documentation for advanced usage of the web interface.
Expand Down Expand Up @@ -65,7 +65,7 @@ $ sourced sql "DESCRIBE TABLE commits;"

## UAST. Parsing code

_Please, refer to the [quick explanation about what Babelfish is](../advanced/bblfsh.md) to know more about it._
_Please, refer to the [quick explanation about what Babelfish is](../usage/bblfsh.md) to know more about it._

You can get UASTs from the `UAST` tab (parsing files by direct input), or using the `UAST` gitbase function over blob contents on `SQL Lab` tab.

Expand Down
12 changes: 6 additions & 6 deletions docs/quickstart/README.md
Expand Up @@ -4,9 +4,9 @@ This guide covers the full setup journey, from zero to populated dashboard with

This process is divided into the following steps:

1. [Install **source{d} CE** dependencies](./1-install-requirements.md),
1. [Install **source{d} CE**](./2-install-sourced.md),
1. [Initialize the dataset](./3-init-sourced.md):
- using local git data,
- using git repositories from your GitHub org,
1. [Explore your dataset](./4-explore-sourced.md),
1. [Install **source{d} CE** Dependencies](./1-install-requirements.md)
1. [Install **source{d} CE**](./2-install-sourced.md)
1. [Initialize the Dataset](./3-init-sourced.md):
- using local git data
- using git repositories from your GitHub org
1. [Explore Your Dataset](./4-explore-sourced.md)
9 changes: 5 additions & 4 deletions docs/usage/README.md
@@ -1,7 +1,8 @@
# Ussage of source{d} Community Edition
# Usage of source{d} Community Edition

Once you know how to install and run **source{d} Community Edition**, you will find in this section some useful resources for guiding your first steps using this tool.

- [Using multiple datasets](./multiple-datasets.md)
- [Some SQL examples to explore your dataset](./examples.md)
- [Available commands in `sourced`](./commands.md)
- [`sourced` Command Reference](./commands.md)
- [Using Multiple Datasets](./multiple-datasets.md)
- [Some SQL Examples to Explore Your Dataset](./examples.md)
- [Babelfish UAST](./bblfsh.md), how to understand code structure
File renamed without changes.
14 changes: 7 additions & 7 deletions docs/usage/commands.md
@@ -1,6 +1,6 @@
# List of `sourced` Sub-Commands

`sourced` binary offers you different kind of sub-commands:
`sourced` binary offers you different kinds of sub-commands:
- [to manage their containers](#manage-containers)
- [to manage **source{d} CE** configuration](#manage-configuration)
- [to open interfaces to access its data](#open-interfaces)
Expand All @@ -10,7 +10,7 @@ Here is the list of all these commands and its description; you can get more inf
adding `--help` when you run it.


## Manage containers
## Manage Containers

### sourced init

Expand All @@ -28,12 +28,12 @@ It can work over a local repository or a list of GitHub organizations.
$ sourced init orgs --token=_USER_TOKEN_ org1,org2...
```

Installs and initializes **source{d} CE** for a list of GitHub organizations, downloading its repositories and
Installs and initializes **source{d} CE** for a list of GitHub organizations, downloading their repositories and
metadata: Users, PullRequests, Issues...

The `orgs` argument must be a comma-separated list of GitHub organization.
The `orgs` argument must be a comma-separated list of GitHub organizations.

The `--token` must contain a valid GitHub user token for the past organizations. It should be granted with
The `--token` must contain a valid GitHub user token for the given organizations. It should be granted with
'repo' and'read:org' scopes.

#### sourced init local
Expand All @@ -54,7 +54,7 @@ Stops all running containers without removing them. They can be started again wi

### sourced prune

Stops containers and removes its containers, networks, volumes, and configurations created by `init` for the current working directory.
Stops containers and removes containers, networks, volumes, and configurations created by `init` for the current working directory.

To delete resources for all the installed working directories, add the `--all` flag.

Expand All @@ -63,7 +63,7 @@ Container images are not deleted unless you specify the `--images` flag.
If you want to completely uninstall `sourced` you must also delete the `~/.sourced` directory.


## Manage configuration
## Manage Configuration

### sourced status

Expand Down
2 changes: 1 addition & 1 deletion docs/usage/examples.md
Expand Up @@ -89,7 +89,7 @@ WHERE

## Queries With UASTs

_**Note**: UAST values are returned as binary blobs; they're best visualized in the `web sql` interface rather than the CLI where are seen as binary data._
_**Note**: UAST values are returned as binary blobs; they're best visualized in the web UI interface rather than the CLI where are seen as binary data._

**Retrieve the UAST for all files at [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):**

Expand Down
2 changes: 1 addition & 1 deletion docs/usage/multiple-datasets.md
Expand Up @@ -34,4 +34,4 @@ You can list all the installed instances, and know which one is active at any mo
If you are familiar with Docker Compose and you want more control over the underlying resources, you can explore the contents of your `~/.sourced` directory. There you will find a `docker-compose.yml` and `.env` files for each set of repositories used by `sourced init`.

_You can read more about how the environments are isolated in the **source{d} CE**
[architecture docs](../advanced/architecture.md)_
[architecture docs](../architecture.md)_

0 comments on commit ee704a9

Please sign in to comment.