# Zotero Integration Guide

**Version**: 1.0.0-rc.4

Complete guide for using ClioDeck's Zotero integration to manage your bibliography and PDFs.

## Overview

ClioDeck integrates with [Zotero](https://www.zotero.org/) to streamline academic writing workflows. This integration allows you to:

- Import bibliographies directly from Zotero collections
- Detect PDF attachments available in your Zotero library
- Download PDFs from Zotero with one click
- Automatically index downloaded PDFs for RAG-based research
- Keep your bibliography in sync with your Zotero library

---

## Prerequisites

### 1. Zotero Account and Desktop Application

- Create a free account at [zotero.org](https://www.zotero.org/)
- Install the Zotero desktop application
- Organize your references in collections
- Ensure your PDFs are attached to citations in Zotero

### 2. Zotero API Credentials (skip if you'll use Local mode instead)

ClioDeck also supports a fully offline **Local mode** that reads your
Zotero library directly from disk — no account, no API key, see the
"Local mode" section below. The credentials below are only needed for API
mode.

You'll need:
- **User ID**: Your numeric Zotero user ID
- **API Key**: A private API key with read access

**To get your credentials:**

1. Log in to [zotero.org](https://www.zotero.org/)
2. Go to **Settings → Feeds/API**
3. Click **"Create new private key"**
4. Set permissions:
   - ✅ Allow library access (read)
   - ✅ Allow file access (read)
   - ❌ Allow write access (not needed)
5. Save the API key (you won't see it again!)
6. Find your User ID in the URL: `https://www.zotero.org/users/YOUR_USER_ID/`

---

## Configuration

### Setting Up Zotero in ClioDeck

1. Open ClioDeck
2. Go to **Settings** (⚙️ icon in sidebar)
3. Expand the **"Zotero Configuration"** section
4. Enter your credentials:
   - **User ID**: Your numeric Zotero user ID
   - **API Key**: Your private API key
5. Click **"Test Connection"** to verify

**If the test succeeds**, you'll see: ✅ "Zotero connection successful"

**If the test fails**, check:
- Your internet connection
- User ID and API Key are correct
- API key has read permissions enabled

### Local mode — no internet, no API key

Everything above describes **API mode**. There is a second, entirely
offline mode: in the same Zotero Configuration section, a mode selector
lets you pick **"Local database (reads zotero.sqlite)"** instead. Point it
at your Zotero data directory (usually `~/Zotero` — it needs to contain
`zotero.sqlite`) and ClioDeck reads your library directly from disk, with
no network call and no API key. Everything else in this guide (import,
sync, PDF handling) works the same way regardless of which mode you pick —
the two are interchangeable data sources behind the same service.

You can keep the Zotero desktop app open while using Local mode: ClioDeck
never opens `zotero.sqlite` in place. It copies the database (and its
`-wal` write-ahead log, if present) to a temporary file and opens that
copy read-only (`ZoteroLocalDB.ts`) — so it doesn't contend with Zotero's
own lock on the live file, and can't corrupt it either.

---

## Importing Bibliography from Zotero

### Method 1: Import Specific Collection

1. Go to **Bibliography** panel (📚 icon in sidebar)
2. Expand **"Zotero Import"** section
3. Configure import options:
   - **Collection**: Select a specific collection (or leave empty for entire library)
   - **☑️ Export BibTeX**: Download bibliography file to project
   - **☐ Download PDFs**: Download all PDFs immediately (optional)
4. Click **"Sync with Zotero"**

**Result**: ClioDeck will import all citations from the selected collection.

### Method 2: Import from BibTeX File

If you've already exported a `.bib` file from Zotero:

1. Go to **Bibliography** panel
2. Click **➕ Import BibTeX** button
3. Select your `.bib` file
4. Choose import mode:
   - **Replace**: Replace entire bibliography (recommended for new projects)
   - **Merge**: Add new citations, skip duplicates

**Merge mode** is useful for updating an existing bibliography without losing manual changes.

---

## Working with PDFs

### PDF Detection

ClioDeck automatically detects PDFs attached to your Zotero citations:

- **📎 Badge**: Shows number of PDFs available in Zotero
- **📄 Badge**: Local PDF file attached to citation
- **✅ Badge**: PDF is indexed and ready for RAG queries

**Example**:
```
Author (2024)  📎 2  ✅
```
This citation has 2 PDFs in Zotero, one is downloaded and indexed locally.

### Downloading Individual PDFs

**To download and index a PDF**:

1. Expand a citation card in the Bibliography panel
2. Click **🔍 Index PDF** button
3. If multiple PDFs are available, select which one to download
4. ClioDeck will:
   - Download the PDF from Zotero
   - Save it to `PDFs/` folder in your project
   - Automatically index it for RAG queries
   - Update the citation with the file path

**Re-indexing**: If you click on an already-indexed PDF, ClioDeck will ask if you want to re-index it (useful after changing chunking configuration).

### Batch Operations

**Download All Missing PDFs** (Added in BETA 2):

If you have many citations with available PDFs in Zotero but not downloaded locally:

1. Go to **Bibliography** panel
2. Click the **📥 Download All** button (appears when PDFs are available)
3. Confirm the batch operation
4. ClioDeck will:
   - Download all missing PDFs from Zotero
   - Automatically index each one
   - Show progress with current citation being processed

**Index All PDFs**:

If you've already downloaded PDFs but haven't indexed them:

1. Go to **Bibliography** panel
2. Click the **📚 Index All** button
3. Confirm the batch operation
4. Wait for indexing to complete

**Progress tracking**: Batch operations show real-time progress with:
- Current citation being processed
- Number completed vs. total
- Success/skip/error counts

---

## Understanding PDF Status

### Status Indicators

| Badge | Meaning | Action Available |
|-------|---------|------------------|
| 📎 2  | 2 PDFs available in Zotero (not downloaded) | Download via Index button |
| 📄    | Local PDF file attached | Index for RAG |
| ✅    | PDF indexed and ready for queries | Re-index if needed |

### Workflow Examples

**New citation with Zotero PDFs**:
```
1. Import from Zotero → Shows 📎 badge
2. Click "Index PDF" → Downloads and indexes → Shows ✅
3. Now you can query this PDF in RAG mode
```

**Citation with local PDF (no Zotero)**:
```
1. Import BibTeX with local PDF paths → Shows 📄
2. Click "Index PDF" → Indexes existing file → Shows ✅
```

**Multiple PDFs available**:
```
1. Citation shows 📎 3
2. Click "Index PDF"
3. Dialog shows all 3 PDFs with filenames
4. Select the one you want
5. ClioDeck downloads and indexes selected PDF
```

---

## Advanced Features

### Merge vs. Replace Import

**Replace mode** (`loadBibliography`):
- Completely replaces your current bibliography
- Use for: Starting fresh, switching projects
- ⚠️ Warning: Deletes all existing citations

**Merge mode** (`mergeBibliography`):
- Adds new citations without removing existing ones
- Detects duplicates by citation key
- Skips duplicates, adds only new citations
- Use for: Updating bibliography, combining sources

**Import summary** shows:
- Total citations after import
- New citations added (merge mode only)
- Duplicates skipped (merge mode only)

### Bibliography Source Tracking

When you import from Zotero or BibTeX, ClioDeck saves the source to `project.json`:

```json
{
  "bibliographySource": {
    "type": "file",
    "filePath": "bibliography.bib"
  }
}
```

The real shape, from `project-manager.ts:51-55`, is narrower than an
earlier version of this page showed — no `userId`, no `lastSync`, no
`collectionKey` — and it's genuinely written after every sync, not a
dead type:

```json
{
  "bibliographySource": {
    "type": "zotero",
    "filePath": "bibliography.bib",
    "zoteroCollection": "ABC123XYZ"
  }
}
```

`ZoteroImport.tsx` writes exactly this after every successful sync
(`window.electron.project.setBibliographySource(...)`). But `type` is
mostly a descriptive tag: when ClioDeck reads this back
(`project-manager.ts`), `'file'` and `'zotero'` sources are treated
identically — only `filePath` (the synced `.bib` file) is actually used to
locate the bibliography.

**Watch out when re-syncing**: `setBibliographySource` overwrites the whole
object rather than merging it, and the collection selector resets to empty
every time you reopen the sync dialog (`useState('')` in
`ZoteroImport.tsx`). If you sync once with a collection selected, then
later re-sync **without** re-picking a collection, `zoteroCollection` gets
silently overwritten with `undefined` — the earlier association is lost,
with no warning. **Fixed in RC4**: the chosen collection is now actually
remembered between syncs.

> **Fixed in RC4.** Until then, starting a sync on project A and switching
> to project B before it finished wrote A's Zotero collections into B's
> vector store: the handler re-read the *currently active* store only after
> the network call resolved. The same pattern existed in the
> `zotero:apply-updates` handler.
>
---

## Best Practices

### 1. Organize Your Zotero Library

Before importing:
- Create collections for different projects
- Ensure citations have complete metadata (author, year, title)
- Attach PDFs to citations in Zotero
- Use consistent citation keys

### 2. Choose Your Workflow

**Option A: Zotero-First Workflow**
- Manage everything in Zotero
- Import specific collections to ClioDeck
- Download only the PDFs you need
- Periodically re-sync to get updates

**Option B: BibTeX Export Workflow**
- Export `.bib` from Zotero
- Import to ClioDeck (replace or merge)
- Manually download missing PDFs
- Re-export from Zotero when needed

**Option C: Hybrid Workflow**
- Import initial bibliography from Zotero
- Download all PDFs at once
- Make local edits in ClioDeck
- Use merge mode to add new citations

### 3. Batch Operations Strategy

**When starting a new project**:
1. Import bibliography from Zotero (without downloading PDFs)
2. Review your citations list
3. Use "Download All Missing PDFs" for complete import
4. Let batch indexing complete (may take time for large libraries)

**When working incrementally**:
1. Import bibliography
2. Download PDFs individually as you need them for research
3. Index on-demand while writing

### 4. Managing Large Libraries

For libraries with 100+ citations:
- Import specific collections rather than entire library
- Download PDFs in batches (50-100 at a time)
- Monitor disk space (PDFs can be large)
- Consider chunking configuration (see RAG settings)

---

## Troubleshooting

### "Zotero connection failed"

**Possible causes**:
- Incorrect User ID or API Key
- API key doesn't have read permissions
- Network connection issues
- Zotero API is down

**Solutions**:
1. Verify credentials in Zotero settings
2. Create a new API key with correct permissions
3. Test connection with `curl`:
   ```bash
   curl "https://api.zotero.org/users/YOUR_USER_ID/collections?key=YOUR_API_KEY"
   ```

### "No PDFs available"

**Possible causes**:
- Citation doesn't have PDF attachments in Zotero
- PDFs are stored locally in Zotero but not synced to cloud
- File storage quota exceeded in Zotero

**Solutions**:
1. Open Zotero desktop app and verify PDF is attached
2. Ensure PDF sync is enabled in Zotero preferences
3. Check your Zotero storage quota
4. Manually attach PDF in Zotero, then re-import

### "PDF download failed"

**Possible causes**:
- Network interruption
- Zotero file sync not enabled
- File permissions in project directory
- Large PDF file (timeout)

**Solutions**:
1. Check internet connection
2. Enable file syncing in Zotero preferences
3. Verify write permissions in project `PDFs/` folder
4. Try downloading individual PDF instead of batch
5. Manually download from Zotero and add via file path

### "Duplicate citations after merge"

**Cause**: Citation keys don't match between sources

**Solution**:
- Use consistent citation key format in Zotero (e.g., `author2024title`)
- Clean up bibliography before merging
- Use Replace mode if you want to start fresh

### Batch operation stuck

**If batch download/index freezes**:
1. Wait 5 minutes (large PDFs take time)
2. Check console for errors (View → Developer → Developer Tools)
3. Restart ClioDeck if necessary
4. Process remaining PDFs individually

---

## PDF Indexing and RAG Configuration

After downloading PDFs, they must be **indexed** before you can query them in RAG mode.

### Indexing Process

**What happens during indexing**:
1. PDF is parsed and text extracted
2. Text is split into chunks (configured in RAG settings)
3. Each chunk is embedded using your configured embedding model (e.g., `nomic-embed-text`)
4. Embeddings are stored in the shared vector database (`.cliodeck/brain.db`)
5. Citation metadata is linked to chunks

**Time required**: Varies by PDF size and system performance
- Small paper (10 pages): ~10-20 seconds
- Large book chapter (50 pages): ~1-2 minutes
- Entire book (300 pages): ~5-10 minutes

### RAG Settings Impact

**Embedding Model** (Settings → LLM, `LLMConfigSection.tsx`):
- ⚠️ **Warning**: Changing embedding model requires re-indexing ALL PDFs
- Embeddings from different models are not compatible
- Recommended: `nomic-embed-text` (768 dim, multilingual)

**Chunking Strategy** (Settings → RAG, `RAGConfigSection.tsx`):
- ⚠️ **Warning**: Changing chunking config requires re-indexing ALL PDFs
- Options:
  - **CPU Optimized**: Small chunks (300 words), faster processing
  - **Standard**: Medium chunks (500 words), balanced
  - **Large**: Big chunks (800 words), more context per chunk

**Best practice**: Configure RAG settings BEFORE batch indexing to avoid re-indexing.

---

## Keyboard Shortcuts

| Action | Shortcut | Notes |
|--------|----------|-------|
| Open Bibliography Panel | Click 📚 in sidebar | |
| Expand/Collapse Citation | Click ▶/▼ | Shows citation details |
| Insert Citation | ✍️ button when expanded | Inserts into active editor |
| Index PDF | 🔍 button when expanded | Downloads if needed |

---

## Related Documentation

- **[Installation Guide](./1.-ClioDeck-Installation.md)**: Setting up ClioDeck
- **[Technical Architecture](./2.-Technical-Architecture.md)**: How Zotero integration works internally
- **[Keyboard Shortcuts](./1.4-Keyboard-Shortcuts.md)**: All available shortcuts

---

## FAQ

**Q: Can I use ClioDeck without Zotero?**
A: Yes! You can import BibTeX files from any source and manually add PDFs.

**Q: Does ClioDeck modify my Zotero library?**
A: No. ClioDeck only reads from Zotero. All changes are local to your project.

**Q: How do I update my bibliography after adding citations in Zotero?**
A: Re-sync from Zotero using merge mode to add new citations without losing local data.

**Q: Can I edit citations in ClioDeck?**
A: Currently no. Edit citations in Zotero, then re-import. Citation editing in ClioDeck is planned for a future release.

**Q: What happens if I delete a PDF from Zotero?**
A: Your local copy remains in the `PDFs/` folder. ClioDeck won't detect the deletion unless you re-sync.

**Q: Can I index PDFs not from Zotero?**
A: Yes! Manually add PDF path to citation in bibliography, then click "Index PDF".

**Q: How much disk space do I need?**
A: Varies by PDF count and size. Estimate:
- 50 papers: ~100-500 MB
- 200 papers: ~500 MB - 2 GB
- Vector database: ~10-50 MB per 100 papers

**Q: Is there a limit to how many PDFs I can index?**
A: No hard limit. Performance depends on your system resources. Tested with 500+ PDFs successfully.

---

## History

The integration started with the Zotero API, collection listing, and BibTeX
export, then gained enhanced PDF detection (attachment badges, multi-PDF
selection, batch "Download All Missing PDFs", automatic download-then-index)
during the pre-1.0 beta period. See the
[RC2](https://github.com/cliodeck/cliodeck-app/wiki/3.2-RC2-Release-Notes)
and [RC3](https://github.com/cliodeck/cliodeck-app/wiki/3.3-RC3-Release-Notes)
release notes for what changed since.

---

## Support

For issues or questions:
- **GitHub Issues**: [github.com/cliodeck/cliodeck-app/issues](https://github.com/cliodeck/cliodeck-app/issues)
- **Wiki**: [github.com/cliodeck/cliodeck-app/wiki](https://github.com/cliodeck/cliodeck-app/wiki)