From 433fcd96b087df704bfb05a6c5d49357291f5346 Mon Sep 17 00:00:00 2001 From: msukkari Date: Tue, 24 Sep 2024 21:52:52 -0700 Subject: [PATCH 01/12] add initial sections for new getting started section in README --- README.md | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/README.md b/README.md index a3eab7221..3b7c0d074 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,66 @@ Sourcebot is a fast code indexing and search tool for your codebases. It is buil # Getting Started +Before getting started, please make sure you install Docker . This is the only dependency required to get started using Sourcebot. + +You can try out Sourcebot on your local machine without cloning the repo by running this simple command: + +``` +docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=sample-config.json ghcr.io/taqlaai/sourcebot:main +``` + +Navigate to `localhost:3000` in your favorite web browser to see Sourcebot in action running on your own machine! This sample command will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please checkout ! + +
+What does this command do? + +- Pull and run the Sourcebot docker image from [ghcr.io/taqlaai/sourcebot:main](ghcr.io/taqlaai/sourcebot:main) +- Set the `CONFIG_PATH` environment variable in the container to `sample-config.json`. Sourcebot loads the config file located at `CONFIG_PATH` to determine which repositories to index. To make things easier to try Sourcebot, we've baked in an [example](https://github.com/TaqlaAI/sourcebot/blob/main/sample-config.json) config file named `sample-config.json` into the published Docker image. +- Map port 3000 between your machine and the docker image (`-p 3000:3000`). This is what allows you to run Sourcebot by navigating to `localhost:3000` +
+ +## Using Sourcebot on a custom repository + +Sourcebot supports indexing and searching through public and private repositories hosted on GitHub, GitLab, and BitBucket. This section will guide you through configure the repositories that Sourcebot indexes. + +### Create a Sourcebot workspace +The Sourcebot workspace is a directory on your machine that stores your Sourcebot config and cache data. To create a Sourcebot workspace, simply create a new directory on your machine (ex. `sourcebot_workspace`): + +### Create a Sourcebot config +Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` from within the workspace you mount. + +From within your Sourcebot workspace, create a new file called `config.json` and paste in the following sample config which loads the `sourcebot` github repo: +``` +{ + "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json", + "Configs": [ + { + "Type": "github", + "GitHubOrg": "TaqlaAI", + "Name": "^sourcebot$" + } + ] +} +``` + +TODO: add instructions here on how to change config +### Mount your workspace and run Sourcebot +Finally, we run Sourcebot but this time we mount the workspace folder so that it can pickup our custom config. + +> [!NOTE] +> This command assumes that you are running it from within your Sourcebot workspace. If you're not, replace `$(pwd)` below with the absolute path of your Sourcebot workspace + +``` +docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main +``` + +### (Optional) Provide access token to index private repositories +TODO +### (Optional) Specifying a specific config file +TODO + + +--- ## Using Docker From 5fc76e41a024ff7c5e0853c4c72eb9979368733d Mon Sep 17 00:00:00 2001 From: msukkari Date: Tue, 24 Sep 2024 21:54:54 -0700 Subject: [PATCH 02/12] add missing section reference --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3b7c0d074..25fc0345e 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ You can try out Sourcebot on your local machine without cloning the repo by runn docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=sample-config.json ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` in your favorite web browser to see Sourcebot in action running on your own machine! This sample command will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please checkout ! +Navigate to `localhost:3000` in your favorite web browser to see Sourcebot in action running on your own machine! This sample command will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please read [Using Sourcebot on a custom repository](#using-sourcebot-on-a-custom-repository) !
What does this command do? From ec8b1dc061fee28922f189822c8ee469c3b63a0c Mon Sep 17 00:00:00 2001 From: msukkari Date: Tue, 24 Sep 2024 21:56:34 -0700 Subject: [PATCH 03/12] simplify english in workspace section --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 25fc0345e..7783e3b37 100644 --- a/README.md +++ b/README.md @@ -51,9 +51,9 @@ Sourcebot supports indexing and searching through public and private repositorie The Sourcebot workspace is a directory on your machine that stores your Sourcebot config and cache data. To create a Sourcebot workspace, simply create a new directory on your machine (ex. `sourcebot_workspace`): ### Create a Sourcebot config -Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` from within the workspace you mount. +Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` within the workspace you mount. -From within your Sourcebot workspace, create a new file called `config.json` and paste in the following sample config which loads the `sourcebot` github repo: +Create a new file called `config.json` inside your workspace and paste in the following sample config which loads the `sourcebot` github repo: ``` { "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json", From 07b46548e398301a3beb1bc70d587d081cfcf655 Mon Sep 17 00:00:00 2001 From: msukkari Date: Tue, 24 Sep 2024 21:59:50 -0700 Subject: [PATCH 04/12] additional language around mounting and running --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7783e3b37..35e3c2352 100644 --- a/README.md +++ b/README.md @@ -72,12 +72,14 @@ TODO: add instructions here on how to change config Finally, we run Sourcebot but this time we mount the workspace folder so that it can pickup our custom config. > [!NOTE] -> This command assumes that you are running it from within your Sourcebot workspace. If you're not, replace `$(pwd)` below with the absolute path of your Sourcebot workspace +> This command assumes that you're running it from within your Sourcebot workspace. If you're not, replace `$(pwd)` below with the absolute path of your Sourcebot workspace ``` docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main ``` +Navigate to `localhost:3000` to use Sourcebot with the custom repository you defined in your config! + ### (Optional) Provide access token to index private repositories TODO ### (Optional) Specifying a specific config file From de0919ebc96b5572baaff96cde072aa19b170fea Mon Sep 17 00:00:00 2001 From: msukkari Date: Tue, 24 Sep 2024 22:01:20 -0700 Subject: [PATCH 05/12] nit, browser language --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 35e3c2352..11bbdc4bc 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ Finally, we run Sourcebot but this time we mount the workspace folder so that it docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` to use Sourcebot with the custom repository you defined in your config! +Navigate to `localhost:3000` in your browser to use Sourcebot with the custom repository you defined in your config! ### (Optional) Provide access token to index private repositories TODO From 002fef8984263f170e0e2cb5bea49d8de89ef050 Mon Sep 17 00:00:00 2001 From: msukkari Date: Wed, 25 Sep 2024 20:22:23 -0700 Subject: [PATCH 06/12] add info on how to edit config file and cleanup old docs --- README.md | 174 ++++++++++++++++++++++-------------------------------- 1 file changed, 69 insertions(+), 105 deletions(-) diff --git a/README.md b/README.md index 11bbdc4bc..521830090 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,10 @@ Sourcebot is a fast code indexing and search tool for your codebases. It is buil # Getting Started +You can try out our public hosted demo [here](https://demo.sourcebot.dev/)! + +#### Running Sourcebot example locally + Before getting started, please make sure you install Docker . This is the only dependency required to get started using Sourcebot. You can try out Sourcebot on your local machine without cloning the repo by running this simple command: @@ -33,7 +37,7 @@ You can try out Sourcebot on your local machine without cloning the repo by runn docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=sample-config.json ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` in your favorite web browser to see Sourcebot in action running on your own machine! This sample command will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please read [Using Sourcebot on a custom repository](#using-sourcebot-on-a-custom-repository) ! +Navigate to `localhost:3000` in your favorite web browser to see Sourcebot running on your own machine! This example will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please read [how to use Sourcebot on a custom repository](#using-sourcebot-on-a-custom-repository) !
What does this command do? @@ -45,13 +49,13 @@ Navigate to `localhost:3000` in your favorite web browser to see Sourcebot in ac ## Using Sourcebot on a custom repository -Sourcebot supports indexing and searching through public and private repositories hosted on GitHub, GitLab, and BitBucket. This section will guide you through configure the repositories that Sourcebot indexes. +Sourcebot supports indexing and searching through public and private repositories hosted on GitHub, GitLab, and BitBucket. This section will guide you through configuring the repositories that Sourcebot indexes. ### Create a Sourcebot workspace The Sourcebot workspace is a directory on your machine that stores your Sourcebot config and cache data. To create a Sourcebot workspace, simply create a new directory on your machine (ex. `sourcebot_workspace`): ### Create a Sourcebot config -Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` within the workspace you mount. +Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` within the mounted workspace. Create a new file called `config.json` inside your workspace and paste in the following sample config which loads the `sourcebot` github repo: ``` @@ -61,15 +65,37 @@ Create a new file called `config.json` inside your workspace and paste in the fo { "Type": "github", "GitHubOrg": "TaqlaAI", - "Name": "^sourcebot$" + "Name": "sourcebot" } ] } ``` -TODO: add instructions here on how to change config +##### Changing the config file +The config file follows the schema defined [here](https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json). You can define multiple repos by adding config objects into the `Configs` list: + +
+{
+    "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json",
+    "Configs": [
+        {
+            "Type": "github",
+            "GitHubOrg": "TaqlaAI", // github orgs must define the GithubOrg field
+            "Name": "sourcebot"     // name of the github repository to index (regex match)
+        },
+        {
+            "Type": "github",
+            "GithubUser": "ssloy",  // if indexing a github repo owned by a user, set the GithubUser field
+            "Name": "tinyrenderer"  // name of the github repository to index (regex match)
+        }
+    ] 
+}
+
+ +TODO: add examples for GitLab and BitBucket above + ### Mount your workspace and run Sourcebot -Finally, we run Sourcebot but this time we mount the workspace folder so that it can pickup our custom config. +We can now run Sourcebot, but this time we mount the workspace folder so that it can pickup our custom config: > [!NOTE] > This command assumes that you're running it from within your Sourcebot workspace. If you're not, replace `$(pwd)` below with the absolute path of your Sourcebot workspace @@ -78,117 +104,54 @@ Finally, we run Sourcebot but this time we mount the workspace folder so that it docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` in your browser to use Sourcebot with the custom repository you defined in your config! - -### (Optional) Provide access token to index private repositories -TODO -### (Optional) Specifying a specific config file -TODO - - ---- -## Using Docker - - -0. Install Docker - -1. Create a `config.json` file and list the repositories you want to index. The JSON schema [index.json](./schemas/index.json) defines the structure of the config file and the available options. For example, if we want to index Sourcebot on its own code, we could use the following config found in `sample-config.json`: - - ```json - { - "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json", - "Configs": [ - { - "Type": "github", - "GitHubOrg": "TaqlaAI", - "Name": "^sourcebot$" - } - ] - } - ``` - -Sourcebot also supports indexing GitLab & BitBucket. Checkout the [index.json](./schemas/index.json) for a full list of available options. +Navigate to `localhost:3000` in your browser to use Sourcebot to search the repositories you've defined in your config! -2. Create a Personal Access Token (PAT) to authenticate with a code host(s): +### (Optional) Provide an access token to index private repositories +In order to allow Sourcebot to index your private repositories, you must provide it with an access token. -
-
- GitHub - - Generate a GitHub Personal Access Token (PAT) [here](https://github.com/settings/tokens/new). If you're only indexing public repositories select the `public_repo` scope; otherwise, select the `repo` scope. - - You'll need to pass this PAT each time you run Sourcebot, so we recommend adding it as an environment variable. In this guide, we'll add the Github PAT as an environment variable called `GITHUB_TOKEN`: - ```sh - export GITHUB_TOKEN= - ``` - - If you'd like to persist this environment variable across shell sessions, please add this line to your shell config file (ex. `~/.bashrc`, `~/.bash_profile`, etc) - - -
- -
- GitLab - - TODO - -
- -
- BitBucket - - TODO +
+
+ GitHub -
-
+Generate a GitHub Personal Access Token (PAT) [here](https://github.com/settings/tokens/new) and make sure you select the `repo` scope. -3. Launch the latest image from the [ghcr registry](https://github.com/TaqlaAI/sourcebot/pkgs/container/sourcebot): +You'll need to pass this PAT each time you run Sourcebot by setting the GITHUB_TOKEN environment variable: -
-
- GitHub +
+docker run -p 3000:3000 --rm --name sourcebot -e GITHUB_TOKEN=[your-github-token] -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main
+
- Run the `sourcebot` docker image, passing in the Github PAT you generated in the previous step as an environment variable called `GITHUB_TOKEN`: - ```sh - docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data -e GITHUB_TOKEN=$GITHUB_TOKEN ghcr.io/taqlaai/sourcebot:main - ``` -
+
-
- GitLab +
+ GitLab - ```sh - docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data -e GITLAB_TOKEN= ghcr.io/taqlaai/sourcebot:main - ``` +TODO -
+
-
- BitBucket +
+ BitBucket - TODO +TODO -
-
+
+ - Two things should happen: (1) a `.sourcebot` directory will be created containing the mirror repositories and indexes, and (2) you will see output similar to: +### (Optional) Specifying a config file - ```sh - INFO spawned: 'node-server' with pid 10 - INFO spawned: 'zoekt-indexserver' with pid 11 - INFO spawned: 'zoekt-webserver' with pid 12 - run [zoekt-mirror-github -dest /data/.sourcebot/repos -delete -org ] - ... - INFO success: node-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) - INFO success: zoekt-indexserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) - INFO success: zoekt-webserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) - ``` +By default Sourcebot will look for a file called `config.json` from within the directory that you [mount to `/data`](#mount-your-workspace-and-run-sourcebot). However, if you have multiple config files and you'd like to specificy a config file you can do so by setting the `CONFIG_PATH` environment variable when running the Docker image: - zoekt will now index your repositories (at `HEAD`). By default, it will re-index existing repositories every hour, and discover new repositories every 24 hours. +
+docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=path/to/your/config_file -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main
+
-4. Go to `http://localhost:3000` - once an index has been created, you can start searching. ## Building Sourcebot +>[!NOTE] +>You don't need to build Sourcebot in order to use it! If you'd just like to use Sourcebot, please read [how to use Sourcebot with custom repositories](#using-sourcebot-on-a-custom-repository). + +If you'd like to make changes to Sourcebot you'll need to build from source: 1. Install go and NodeJS. Note that a NodeJS version of at least `21.1.0` is required. @@ -218,7 +181,7 @@ The zoekt binaries and web dependencies are placed into `bin` and `node_modules` { "Type": "github", "GitHubOrg": "TaqlaAI", - "Name": "^sourcebot$" + "Name": "sourcebot" } ] } @@ -262,14 +225,15 @@ The zoekt binaries and web dependencies are placed into `bin` and `node_modules` ## Telemetry -By default, Sourcebot collects anonymized usage data through [PostHog](https://posthog.com/) to help us improve the performance and reliability of our tool. We do not collect or transmit [any information related to your codebase](https://github.com/search?q=repo:TaqlaAI/sourcebot++captureEvent&type=code). All events are [sanitized](https://github.com/TaqlaAI/sourcebot/blob/main/src/app/posthogProvider.tsx) to ensure that no sensitive or identifying details leave your machine. The data we collect includes general usage statistics and metadata such as query performance (e.g., search duration, error rates) to monitor the application's health and functionality. This information helps us better understand how Sourcebot is used and where improvements can be made :) +By default, Sourcebot collects anonymized usage data through [PostHog](https://posthog.com/) to help us improve the performance and reliability of our tool. We do not collect or transmit [any information related to your codebase](https://github.com/search?q=repo:TaqlaAI/sourcebot++captureEvent&type=code). In addition, all events are [sanitized](https://github.com/TaqlaAI/sourcebot/blob/main/src/app/posthogProvider.tsx) to ensure that no sensitive or identifying details leave your machine. The data we collect includes general usage statistics and metadata such as query performance (e.g., search duration, error rates) to monitor the application's health and functionality. This information helps us better understand how Sourcebot is used and where improvements can be made :) If you'd like to disable all telemetry, you can do so by setting the environment variable `SOURCEBOT_TELEMETRY_DISABLED` to `1` in the docker run command: -```sh -docker run -e SOURCEBOT_TELEMETRY_DISABLED=1 /* additional args */ ghcr.io/taqlaai/sourcebot:main -``` -Or if you are building locally, add the following to your [.env](./.env) file: +
+docker run -e SOURCEBOT_TELEMETRY_DISABLED=1 /* additional args */ ghcr.io/taqlaai/sourcebot:main
+
+ +Or if you are [building locally](#building-sourcebot), add the following to your [.env](./.env) file: ```sh SOURCEBOT_TELEMETRY_DISABLED=1 NEXT_PUBLIC_SOURCEBOT_TELEMETRY_DISABLED=1 From 7f809fb419b766ee517861a19f640d43bb423f52 Mon Sep 17 00:00:00 2001 From: msukkari Date: Wed, 25 Sep 2024 20:41:34 -0700 Subject: [PATCH 07/12] add features section --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 7dc7bd11e..389d2adc3 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,13 @@ Sourcebot is a fast code indexing and search tool for your codebases. It is buil ![Demo video](https://github.com/user-attachments/assets/227176d8-fc61-42a9-8746-3cbc831f09e4) +## Features +- 💻 **One-command deployment**: Get started instantly using Docker on your own machine. +- 🔍 **Multi-repo search**: Effortlessly index and search through multiple public and private repositories (GitHub, GitLab, BitBucket). +- ⚡**Lightning fast performance**: Built on top of the powerful [Zoekt](https://github.com/sourcegraph/zoekt) search engine. +- 📂 **Full file visualization**: Instantly view the entire file when selecting any search result. +- 🎨 **Modern web application**: Enjoy a sleek interface with features like syntax highlighting, light/dark mode, and vim-style navigation + # Getting Started You can try out our public hosted demo [here](https://demo.sourcebot.dev/)! From 89b9ea60a03a320204e09404235d9e0fdef6d672 Mon Sep 17 00:00:00 2001 From: msukkari Date: Thu, 26 Sep 2024 11:14:55 -0700 Subject: [PATCH 08/12] made readme more concise --- README.md | 73 ++++++++++++++----------------------------------------- 1 file changed, 18 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 389d2adc3..03d4624da 100644 --- a/README.md +++ b/README.md @@ -30,43 +30,41 @@ Sourcebot is a fast code indexing and search tool for your codebases. It is buil - 📂 **Full file visualization**: Instantly view the entire file when selecting any search result. - 🎨 **Modern web application**: Enjoy a sleek interface with features like syntax highlighting, light/dark mode, and vim-style navigation -# Getting Started - You can try out our public hosted demo [here](https://demo.sourcebot.dev/)! -#### Running Sourcebot example locally - -Before getting started, please make sure you install Docker . This is the only dependency required to get started using Sourcebot. +# Getting Started -You can try out Sourcebot on your local machine without cloning the repo by running this simple command: +Get started with a single docker command: ``` docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=sample-config.json ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` in your favorite web browser to see Sourcebot running on your own machine! This example will index the Sourcebot repository and allow you to search through it. If you'd like to run Sourcebot on a different repository, please read [how to use Sourcebot on a custom repository](#using-sourcebot-on-a-custom-repository) ! +Navigate to `localhost:3000` to start searching the Sourcebot repo. Want to search your own repos? Checkout how to [configure Sourcebot](#configuring-sourcebot).
What does this command do? -- Pull and run the Sourcebot docker image from [ghcr.io/taqlaai/sourcebot:main](ghcr.io/taqlaai/sourcebot:main) +- Pull and run the Sourcebot docker image from [ghcr.io/taqlaai/sourcebot:main](ghcr.io/taqlaai/sourcebot:main). You'll need to make sure you have [docker installed](https://docs.docker.com/get-started/get-docker/) to do this. - Set the `CONFIG_PATH` environment variable in the container to `sample-config.json`. Sourcebot loads the config file located at `CONFIG_PATH` to determine which repositories to index. To make things easier to try Sourcebot, we've baked in an [example](https://github.com/TaqlaAI/sourcebot/blob/main/sample-config.json) config file named `sample-config.json` into the published Docker image. -- Map port 3000 between your machine and the docker image (`-p 3000:3000`). This is what allows you to run Sourcebot by navigating to `localhost:3000` +- Map port 3000 between your machine and the docker image (`-p 3000:3000`).
-## Using Sourcebot on a custom repository +## Configuring Sourcebot Sourcebot supports indexing and searching through public and private repositories hosted on GitHub, GitLab, and BitBucket. This section will guide you through configuring the repositories that Sourcebot indexes. -### Create a Sourcebot workspace -The Sourcebot workspace is a directory on your machine that stores your Sourcebot config and cache data. To create a Sourcebot workspace, simply create a new directory on your machine (ex. `sourcebot_workspace`): +1. Create a new folder on your machine that stores your configs and `.sourcebot` cache, and navigate into it: +``` +mkdir sourcebot_workspace +cd sourcebot_workspace +``` -### Create a Sourcebot config -Sourcebot needs a config file to tell it which repositories to index. By default, Sourcebot will look for a file called `config.json` within the mounted workspace. +2. Create a new config following the [configuration schema](https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json) to specify which repositories Sourcebot should index. For example to index Sourcebot itself: -Create a new file called `config.json` inside your workspace and paste in the following sample config which loads the `sourcebot` github repo: ``` -{ +touch my_config.json +echo `{ "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json", "Configs": [ { @@ -75,43 +73,16 @@ Create a new file called `config.json` inside your workspace and paste in the fo "Name": "sourcebot" } ] -} +}` > my_config.json ``` -##### Changing the config file -The config file follows the schema defined [here](https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json). You can define multiple repos by adding config objects into the `Configs` list: - -
-{
-    "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json",
-    "Configs": [
-        {
-            "Type": "github",
-            "GitHubOrg": "TaqlaAI", // github orgs must define the GithubOrg field
-            "Name": "sourcebot"     // name of the github repository to index (regex match)
-        },
-        {
-            "Type": "github",
-            "GithubUser": "ssloy",  // if indexing a github repo owned by a user, set the GithubUser field
-            "Name": "tinyrenderer"  // name of the github repository to index (regex match)
-        }
-    ] 
-}
-
- -TODO: add examples for GitLab and BitBucket above - -### Mount your workspace and run Sourcebot -We can now run Sourcebot, but this time we mount the workspace folder so that it can pickup our custom config: - -> [!NOTE] -> This command assumes that you're running it from within your Sourcebot workspace. If you're not, replace `$(pwd)` below with the absolute path of your Sourcebot workspace +3. Run Sourcebot and point it to the new config you created: ``` -docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main +docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=./my_config.json -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main ``` -Navigate to `localhost:3000` in your browser to use Sourcebot to search the repositories you've defined in your config! +This command will also mount the current directory (`-v $(pwd):/data`) to allow Sourcebot to persist the `.sourcebot` cache. ### (Optional) Provide an access token to index private repositories In order to allow Sourcebot to index your private repositories, you must provide it with an access token. @@ -145,14 +116,6 @@ TODO
-### (Optional) Specifying a config file - -By default Sourcebot will look for a file called `config.json` from within the directory that you [mount to `/data`](#mount-your-workspace-and-run-sourcebot). However, if you have multiple config files and you'd like to specificy a config file you can do so by setting the `CONFIG_PATH` environment variable when running the Docker image: - -
-docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=path/to/your/config_file -v $(pwd):/data ghcr.io/taqlaai/sourcebot:main
-
- ## Building Sourcebot >[!NOTE] From ad75b11008d00a06bd0c3d4f315f150cb625aafc Mon Sep 17 00:00:00 2001 From: msukkari Date: Thu, 26 Sep 2024 11:19:31 -0700 Subject: [PATCH 09/12] fix github icon --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 03d4624da..ccbbdfcff 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,12 @@ In order to allow Sourcebot to index your private repositories, you must provide
- GitHub + + + + GitHub icon + GitHub + Generate a GitHub Personal Access Token (PAT) [here](https://github.com/settings/tokens/new) and make sure you select the `repo` scope. From 6c5de039ff4a0fb03307bf8dcdc171c626c29c11 Mon Sep 17 00:00:00 2001 From: msukkari Date: Thu, 26 Sep 2024 11:21:03 -0700 Subject: [PATCH 10/12] missed a github icon --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ccbbdfcff..72bb90194 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,13 @@ Navigate to `localhost:3000` to start searching the Sourcebot repo. Want to sear ## Configuring Sourcebot -Sourcebot supports indexing and searching through public and private repositories hosted on GitHub, GitLab, and BitBucket. This section will guide you through configuring the repositories that Sourcebot indexes. +Sourcebot supports indexing and searching through public and private repositories hosted on + + + GitHub icon + GitHub, +GitLab, and + BitBucket. This section will guide you through configuring the repositories that Sourcebot indexes. 1. Create a new folder on your machine that stores your configs and `.sourcebot` cache, and navigate into it: ``` From 05030a7ae4dd61db4024f194e82cdd4ee9603724 Mon Sep 17 00:00:00 2001 From: msukkari Date: Thu, 26 Sep 2024 11:23:49 -0700 Subject: [PATCH 11/12] fixed links in build section --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 72bb90194..5d55ecb2c 100644 --- a/README.md +++ b/README.md @@ -128,9 +128,9 @@ TODO
-## Building Sourcebot +## Build from source >[!NOTE] ->You don't need to build Sourcebot in order to use it! If you'd just like to use Sourcebot, please read [how to use Sourcebot with custom repositories](#using-sourcebot-on-a-custom-repository). +>You don't need to build Sourcebot in order to use it! If you'd just like to use Sourcebot, please read [how to configure Sourcebot](#configuring-sourcebot). If you'd like to make changes to Sourcebot you'll need to build from source: From 95e094ee923304da3a2f83679a0a42d5a325d0d8 Mon Sep 17 00:00:00 2001 From: msukkari Date: Thu, 26 Sep 2024 14:05:20 -0700 Subject: [PATCH 12/12] remove CONFIG_PATH setting in one command instruction: --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5d55ecb2c..7dcd9d6fb 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ You can try out our public hosted demo [here](https://demo.sourcebot.dev/)! Get started with a single docker command: ``` -docker run -p 3000:3000 --rm --name sourcebot -e CONFIG_PATH=sample-config.json ghcr.io/taqlaai/sourcebot:main +docker run -p 3000:3000 --rm --name sourcebot ghcr.io/taqlaai/sourcebot:main ``` Navigate to `localhost:3000` to start searching the Sourcebot repo. Want to search your own repos? Checkout how to [configure Sourcebot](#configuring-sourcebot). @@ -45,8 +45,8 @@ Navigate to `localhost:3000` to start searching the Sourcebot repo. Want to sear
What does this command do? -- Pull and run the Sourcebot docker image from [ghcr.io/taqlaai/sourcebot:main](ghcr.io/taqlaai/sourcebot:main). You'll need to make sure you have [docker installed](https://docs.docker.com/get-started/get-docker/) to do this. -- Set the `CONFIG_PATH` environment variable in the container to `sample-config.json`. Sourcebot loads the config file located at `CONFIG_PATH` to determine which repositories to index. To make things easier to try Sourcebot, we've baked in an [example](https://github.com/TaqlaAI/sourcebot/blob/main/sample-config.json) config file named `sample-config.json` into the published Docker image. +- Pull and run the Sourcebot docker image from [ghcr.io/taqlaai/sourcebot:main](https://github.com/taqlaai/sourcebot/pkgs/container/sourcebot). You'll need to make sure you have [docker installed](https://docs.docker.com/get-started/get-docker/) to do this. +- Sourcebot will index itself to prepare for your search request. - Map port 3000 between your machine and the docker image (`-p 3000:3000`).