Skip to content
BranchesTags

Latest commit

 

History

52 Commits

Folders and files

NameName
Last commit message
Last commit date
shard
shard
 
 
.gitignore
.gitignore
 
 
.python-version
.python-version
 
 
CHANGELOG.md
CHANGELOG.md
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 
uv.lock
uv.lock
 
 

Repository files navigation

Shard — AI-Powered Note Ingestion for Obsidian

License: MIT Python 3.11+ Ollama

Shard ingests PDFs, URLs, YouTube videos, and text into your Obsidian vault as structured AI-generated notes with semantic search. Everything runs locally by default — your data never leaves your machine.

✨ Features

  • 📄 Multi-source ingestion (PDF, URL, YouTube, text, stdin)
  • 🤖 AI-powered formatting with auto-generated titles, tags, summaries
  • 🔍 Semantic search with Redis Stack vector embeddings
  • 🏠 Local-first with Ollama — no cloud, no API keys, no cost
  • 📝 Native Obsidian integration with YAML frontmatter
  • 🔌 Flexible model support via LiteLLM (Ollama, OpenAI, Anthropic, Groq, etc.)
  • ⚡ Fuzzy note search and one-click Obsidian opening
  • 🧠 Learn your style — analyzes your vault and writes new notes that match how you already write
  • ⚛️ Atomic notes — automatically splits any source into focused single-concept notes, all interlinked with [[wikilinks]]
  • 🔗 Auto backlinks — syncs [[wikilinks]] across your vault to build a rich knowledge graph
  • 📁 Flat file saving — notes save directly to your vault root, no buried subfolders
  • 🤖 Model switcher — switch between local and cloud models, pull Ollama models, and manage API keys with shard model

⚡ Quick Start

git clone https://github.com/RicePollution/shard
cd shard
uv tool install .
shard config
shard add "https://example.com/article"
shard ask "what did I just read?"

Need Python, uv, or Ollama? See Prerequisites.

⚙️ First Time Setup

Run:

shard config

Example wizard output:

Note: Default path requires a space for /Documents/Obsidian Vault

Welcome to Shard! Let's get you set up.

Path to your Obsidian vault [/home/user/Documents/ObsidianVault]: /home/user/Documents/MyVault

Checking for a local Ollama installation…
Ollama is running. Found 1 installed model(s).

Installed models:
  qwen2.5:3b

'qwen2.5:3b' is already installed. Using it as the default model.

Config saved to /home/user/.config/shard/config.json

Config file location:

OS Path
🐧 Linux ~/.config/shard/config.json
🍎 macOS ~/.config/shard/config.json
🪟 Windows %APPDATA%\shard\config.json

📖 Commands

shard add

Ingest any source into your vault as atomic notes.

Flag Description
--single One note, no splitting
--instruction, -i Custom formatting instruction (e.g. "focus on code examples") 
shard add "https://example.com"
shard add /path/to/paper.pdf
shard add "https://youtube.com/watch?v=..."
shard add "raw text or idea"
cat file.txt | shard add -
shard add "https://example.com" -i "focus on practical examples"

shard ask

Semantic search across your entire vault with AI answers.

Flag Description
--top-k N Number of source chunks to retrieve (default: 5) 
shard ask "what do I know about RAG?"
shard ask "best practices for testing" --top-k 10

shard model

Manage models and API keys.

Subcommand Description
shard model Interactive model menu
shard model list Show all available models
shard model use <model> Switch to a model
shard model pull <model> Pull an Ollama model
shard model key <provider> Add a cloud API key
shard model key --list Show configured keys 
shard model use gpt-5
shard model pull llama3.1:8b
shard model key openai

shard learn

Learn your vault's writing style for better note generation.

Flag Description
--depth quick 5 notes, 1 API call, fastest
--depth normal 20 notes, balanced (default)
--depth deep Entire vault, most accurate
--force Re-analyze even if profile exists
--show Print current style fingerprint
--template Print blank note template 
shard learn
shard learn --depth deep

shard sync

Add [[wikilinks]] between related notes automatically.

Flag Description
--dry-run Preview changes without modifying files
--verbose Show each link as it's added 
shard sync
shard sync --dry-run

shard index

Rebuild the semantic search index from your vault.

shard index

shard list

Show all imported notes.

shard list

shard open

Fuzzy-match and open a note in Obsidian.

shard open "machine learning"

shard config

View and update settings.

Flag Description
--show Print current config
--set KEY=VALUE Update a single value
--setup Re-run setup wizard 
shard config --show
shard config --set vault_path="/new/path"

🤖 Models

🏷️ Model Tiers

Tier Models Cost RAM Required
🟢 Local Small qwen2.5:3b, phi3.5, llama3.2:3b Free ~4 GB
🟡 Local Large llama3.1:8b, qwen2.5:14b Free 8+ GB
🔵 Cloud Claude, GPT-4o, Gemini, Groq Paid / Free tier N/A

☁️ Cloud Providers

Provider Model String Free Tier API Key
Groq groq/llama-3.1-8b-instant ✅ Yes console.groq.com
OpenAI gpt-4o ❌ No platform.openai.com
Anthropic claude-sonnet-4-20250514 ❌ No console.anthropic.com
Google gemini/gemini-pro ✅ Yes aistudio.google.com

💡 Groq offers a generous free tier with fast inference. It's the best starting point if you want cloud models without paying.

Add API keys with: shard model key <provider>

📋 Prerequisites

🐍 Python 3.11+

Python is the programming language Shard is built with. You need version 3.11 or newer.

Check if installed:

python3 --version
🐧 Arch Linux 
sudo pacman -S python
🐧 Ubuntu / Debian 
sudo apt update
sudo apt install python3
🐧 Fedora / RHEL 
sudo dnf install python3
🍎 macOS 
brew install python3
🪟 Windows

Download the installer from python.org and run it. Make sure to check "Add Python to PATH" during installation.

Verify:

python3 --version  # Must show 3.11 or higher

📦 uv

uv is a modern Python package manager. It's faster than pip and lets you install Python apps as global commands without managing virtual environments. Think of it like installing an app, not a library.

Check if installed:

uv --version
🐧 Arch Linux 
sudo pacman -S uv
🐧 Ubuntu / Debian 
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
🐧 Fedora / RHEL 
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
🍎 macOS 
brew install uv

Or via the installer:

curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.zshrc
🪟 Windows 
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Verify:

uv --version

🦙 Ollama

Ollama runs AI models locally on your machine for free. No account, no API key, no internet needed after setup. Shard uses it by default so your notes stay completely private.

Check if installed:

ollama --version
🐧 Arch Linux 
sudo pacman -S ollama
🐧 Ubuntu / Debian 
curl -fsSL https://ollama.ai/install.sh | sh
🐧 Fedora / RHEL 
curl -fsSL https://ollama.ai/install.sh | sh
🍎 macOS 
brew install ollama

Or download from ollama.com.

🪟 Windows

Download the installer from ollama.com and run it.

Start Ollama:

🐧 Arch Linux 
ollama serve

Or for automatic startup on boot (recommended):

sudo systemctl enable --now ollama
🐧 Ubuntu / Debian 
ollama serve

Or for automatic startup on boot (recommended):

sudo systemctl enable --now ollama
🐧 Fedora / RHEL 
ollama serve

Or for automatic startup on boot (recommended):

sudo systemctl enable --now ollama
🍎 macOS

Ollama runs automatically as a menu bar app after installation. No extra steps needed.

🪟 Windows

Ollama runs automatically as a system tray app after installation. No extra steps needed.

💡 ollama serve must be running whenever you use Shard. On Linux, using systemctl is the easiest option — it starts automatically on boot.

Pull the default model:

ollama pull qwen2.5:3b

💡 This downloads about 2 GB. It runs on any modern machine with no GPU required. Completely free, forever.

Verify:

ollama run qwen2.5:3b "say hello"

🗄️ Redis Stack

Redis Stack provides the vector search engine for shard ask. It stores note embeddings and runs fast similarity queries.

Check if installed:

redis-cli ping
🐧 Arch Linux 
yay -S redis-stack-server
sudo systemctl enable --now redis-stack-server
🐧 Ubuntu / Debian 
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt update
sudo apt install redis-stack-server
sudo systemctl enable --now redis-stack-server
🍎 macOS 
brew tap redis-stack/redis-stack
brew install redis-stack-server
redis-stack-server --daemonize yes
🪟 Windows

Use Docker:

docker run -d --name redis-stack -p 6379:6379 redis/redis-stack-server:latest
🐳 Docker (any OS) 
docker run -d --name redis-stack -p 6379:6379 redis/redis-stack-server:latest

Verify:

redis-cli ping  # Should print PONG

💡 Redis Stack must be running whenever you use shard ask or shard index. The shard add command indexes notes automatically.

🗃️ Obsidian Vault

Obsidian is a note-taking app that stores notes as plain markdown files in a folder called a "vault." Shard saves generated notes directly into your vault.

🐧 Arch Linux 
yay -S obsidian

Or download the AppImage from obsidian.md.

🐧 Ubuntu / Debian

Download the .deb file from obsidian.md and install it:

sudo apt install ./Obsidian-*.deb
🐧 Fedora / RHEL

Download the .rpm file from obsidian.md and install it:

sudo dnf install ./Obsidian-*.rpm
🍎 macOS 
brew install --cask obsidian

Or download from obsidian.md.

🪟 Windows

Download the installer from obsidian.md and run it.

Open Obsidian and create a vault if you haven't. Note the vault folder path — you'll need it during Shard setup. Example paths:

  • Linux: /home/yourname/Documents/MyVault
  • macOS: /Users/yourname/Documents/MyVault
  • Windows: C:\Users\yourname\Documents\MyVault

🚀 Installation

git clone https://github.com/RicePollution/shard
cd shard
uv tool install .

💡 uv tool install . installs Shard as a global command from the local folder. You only need to do this once. After this, shard works from any directory.

If shard is not found:

🐧 Arch Linux 
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
🐧 Ubuntu / Debian 
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
🐧 Fedora / RHEL 
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
🍎 macOS 
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc
🪟 Windows
  1. Press Win + X and select "System"
  2. Click "Advanced system settings" on the left
  3. Click "Environment Variables" button
  4. Under "User variables", click "New"
  5. Variable name: PATH
  6. Variable value: %USERPROFILE%\.local\bin
  7. Click OK on all dialogs
  8. Restart your terminal

Verify:

shard --help

⚙️ Configuration Reference

Field Default Description
vault_path (set during setup) Absolute path to your Obsidian vault
model ollama_chat/qwen2.5:3b LiteLLM model string for note generation
redis_host localhost Redis Stack hostname
redis_port 6379 Redis Stack port
redis_password "" Redis password (if set)
embedding_model all-MiniLM-L6-v2 Sentence-transformers model for embeddings
custom_models [] User-registered model descriptors
api_keys {} Provider API keys (alternative to env vars)
notes_subfolder "" (vault root) Where new notes are saved
style_profile auto-managed Path to learned style JSON

🏗️ How It Works

Your Input → Extractor → Formatter (AI) → Indexer → Obsidian Vault
  • Extractor: Pulls text from PDFs (pdfplumber), URLs (httpx + BeautifulSoup), YouTube (transcript API), or stdin
  • Formatter: Sends text to your LLM to generate a structured note with title, tags, summary, and markdown body
  • Indexer: Chunks the note text, generates vector embeddings (sentence-transformers), and stores them in Redis Stack for semantic search
  • Vault: Saves the note as a markdown file with YAML frontmatter in your Obsidian vault

🔧 Troubleshooting

❌ shard: command not found

Your PATH doesn't include the uv tool install directory.

🐧 Linux:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

🍎 macOS:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

🪟 Windows:

  1. Press Win + X and select "System"
  2. Click "Advanced system settings" on the left
  3. Click "Environment Variables" button
  4. Under "User variables", click "New"
  5. Variable name: PATH
  6. Variable value: %USERPROFILE%\.local\bin
  7. Click OK on all dialogs
  8. Restart your terminal
❌ Connection refused / Cannot reach Ollama

Ollama isn't running. Start it:

🐧 Linux:

ollama serve

Or for automatic startup:

sudo systemctl enable --now ollama

🍎 macOS:

Open the Ollama app from your Applications folder.

🪟 Windows:

Open the Ollama app from your Start Menu.

❌ Model not found

Pull the default model:

ollama pull qwen2.5:3b
❌ Redis is not available

Redis Stack isn't running or doesn't have the RediSearch module loaded.

Start Redis Stack:

# Linux (systemd)
sudo systemctl start redis-stack-server

# macOS (Homebrew)
redis-stack-server --daemonize yes

# Docker
docker start redis-stack

If you have plain Redis without the Stack modules, you need Redis Stack instead. See the Redis Stack prerequisite.

❌ Vault path does not exist

Re-run setup:

shard config --setup
⏳ Slow on first run

Normal! First run loads the AI model into memory and downloads the embedding model (~80 MB). Subsequent runs are much faster.

❌ Permission denied (Linux)

If you get permission errors on Linux:

chmod +x ~/.local/bin/shard
❌ shard learn says not enough notes

shard learn needs at least 5 notes in your vault to analyze. Add some notes first with shard add, then re-run shard learn.

❌ shard sync changed something it shouldn't have

shard sync always creates a backup before making changes. Find your backup at ~/.shard/backups/ and restore from there.

To preview changes without modifying files, always use:

shard sync --dry-run

🤝 Contributing

We welcome contributions! For guidelines, see CONTRIBUTING.md.

Good first issues:

  • Add support for EPUB files
  • Add support for RSS feeds
  • Improve chunking with semantic boundaries
  • Add shard export command
  • Better error messages for common failures

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

About

No description, website, or topics provided.

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages