# BETA 2 Release Notes

Release date: January 2024

## Overview

BETA 2 focuses on **enhanced Zotero integration** and **improved user experience**. This release streamlines the workflow from bibliography import to PDF indexing, making it easier to build a comprehensive research database.

**Key improvements:**
- Seamless PDF download from Zotero
- Batch operations for downloading and indexing
- Better visual indicators and warnings
- Enhanced theme system with automatic switching
- Improved configuration experience

---

## 🎯 Highlights

### 1. Enhanced Zotero PDF Workflow

The complete workflow from Zotero to indexed PDFs is now seamless:

**Before BETA 2:**
```
Import bibliography → Manually find PDFs → Manually add → Index individually
```

**After BETA 2:**
```
Import bibliography → Click "Download All" → Automatic indexing ✅
```

### 2. Visual PDF Status Indicators

New badges show PDF availability at a glance:
- **📎 N**: N PDFs available in Zotero (not downloaded yet)
- **📄**: Local PDF file attached
- **✅**: PDF fully indexed and ready for RAG queries

### 3. Batch Operations

Process multiple PDFs efficiently:
- **Download All Missing PDFs**: Download and index all PDFs from Zotero in one operation
- **Index All PDFs**: Index all local PDFs with progress tracking
- Real-time progress display with current citation being processed

---

## ✨ New Features

### Phase 1: UI & Theme Improvements

#### Automatic Theme Switching
- **Light/Dark/Auto modes**: Choose your preference or let ClioDeck adapt
- **Time-based auto-switching**:
  - Dark mode: 8 PM - 6 AM
  - Light mode: 6 AM - 8 PM
- **Removed**: Dense theme (replaced by better spacing in light/dark themes)

**Configuration**: Settings → Appearance Settings → Theme

#### Topic Modeling Status Fix
- **Fixed**: Status now correctly shows "Not Installed" or "Installed"
- **Improved**: Integrated into CollapsibleSection for consistency
- **Better UX**: Clear installation status and error messages

#### Open Project in Finder/Explorer
- **New button**: Quickly access project folder from Projects panel
- **Platform support**: Finder on macOS, File Explorer on Windows, file manager on Linux
- **Location**: Projects panel → ... → "Open in Finder/Explorer"

### Phase 2: Zotero PDF Detection

#### Extended Citation Type System
- **Zotero metadata**: Citations now include `zoteroAttachments` array
- **Attachment info**: Each attachment includes:
  - `key`: Zotero attachment key
  - `filename`: Original filename
  - `contentType`: MIME type (e.g., `application/pdf`)

**Implementation**: [src/renderer/src/stores/bibliographyStore.ts](https://github.com/cliodeck/cliodeck-app/blob/beta2/src/renderer/src/stores/bibliographyStore.ts)

#### Visual Indicators
- **📎 Badge**: Shows count of available Zotero PDFs
- **Tooltip**: Displays "N PDFs available in Zotero"
- **Color coding**: Visual distinction between indexed/not indexed

**UI Component**: [src/renderer/src/components/Bibliography/CitationCard.tsx](https://github.com/cliodeck/cliodeck-app/blob/beta2/src/renderer/src/components/Bibliography/CitationCard.tsx)

### Phase 3: PDF Download & Indexation

#### Multi-PDF Selection Dialog
When a citation has multiple PDFs attached:
1. Click "Index PDF" button
2. Dialog shows all available PDFs with filenames
3. Select the PDF you want
4. ClioDeck downloads and indexes automatically

**Component**: [src/renderer/src/components/Bibliography/PDFSelectionDialog.tsx](https://github.com/cliodeck/cliodeck-app/blob/beta2/src/renderer/src/components/Bibliography/PDFSelectionDialog.tsx)

#### Download-on-Index Workflow
- **Smart detection**: Checks if PDF is local or needs downloading
- **Automatic download**: If PDF is in Zotero but not local, downloads first
- **Then indexes**: Seamlessly proceeds to indexing after download
- **Progress feedback**: Shows current status (downloading → indexing → done)

#### Batch Download All Missing PDFs
- **New button**: 📥 Download All in Bibliography panel toolbar
- **Batch processing**: Downloads and indexes all citations with Zotero PDFs
- **Progress tracking**:
  - Current citation being processed
  - N / Total counter
  - Success/skipped/error counts
- **Error handling**: Continues processing on errors, reports at the end

**Store method**: `downloadAllMissingPDFs(projectPath: string)`

#### Improved Batch Indexing Display
- **Enhanced UI**: Shows current citation title during batch operations
- **Real-time stats**: ✅ indexed, ⏭️ skipped
- **Progress bar**: Visual completion indicator
- **Responsive**: UI remains interactive during operations

### Phase 4: Configuration Improvements

#### Enhanced Embedding Model Warning
When changing the embedding model in Settings:

```
⚠️ Attention: Changer ce modèle nécessite de ré-indexer tous les PDFs.

Les embeddings ne sont pas compatibles entre modèles différents.
Vous devrez supprimer tous les PDFs indexés et les ré-indexer après le changement.

Modèles recommandés:
• nomic-embed-text - 768 dim, multilingue, recommandé
• mxbai-embed-large - 1024 dim, très performant
• all-minilm - 384 dim, léger et rapide
```

**Location**: Settings → Configuration LLM → Modèle d'embedding

#### Enhanced Chunking Configuration Warning
When changing chunking strategy:

```
⚠️ Attention: Changer cette option nécessite de ré-indexer tous les PDFs.

Les chunks existants ne seront pas mis à jour automatiquement.
```

**Location**: Settings → Configuration RAG → Stratégie de découpage

#### Better UX for Critical Settings
- **Prominent warnings**: Yellow warning boxes that can't be missed
- **Clear explanations**: Why re-indexing is necessary
- **Recommended values**: Guidance on best practices
- **Consistency**: Same warning style across all critical settings

---

## 🔧 Technical Improvements

### Backend Services

#### Zotero PDF Download Service
- **New endpoint**: `window.electron.zotero.downloadPDF()`
- **Parameters**:
  - `userId`, `apiKey`: Zotero credentials
  - `attachmentKey`: Specific attachment to download
  - `filename`: Target filename
  - `targetDirectory`: Project path
- **Returns**: Success status and file path
- **Error handling**: Graceful failures with detailed error messages

**Implementation**: [src/main/services/zotero-service.ts](https://github.com/cliodeck/cliodeck-app/blob/beta2/src/main/services/zotero-service.ts)

#### Enhanced Bibliography Store
New methods added to Zustand store:
- `downloadAllMissingPDFs()`: Batch download and index
- `downloadAndIndexZoteroPDF()`: Download single PDF and index
- `refreshIndexedPDFs()`: Update indexed status cache
- `isFileIndexed()`: Check if PDF is already indexed

**State management**: [src/renderer/src/stores/bibliographyStore.ts](https://github.com/cliodeck/cliodeck-app/blob/beta2/src/renderer/src/stores/bibliographyStore.ts)

### Internationalization

New translation keys for BETA 2 features:
- `bibliography.downloadAllMissing`: "Download all missing PDFs from Zotero"
- `bibliography.confirmDownloadAll`: Confirmation dialog
- `bibliography.downloadAllComplete`: Success message with stats
- `bibliography.pdfDownloadedAndIndexed`: Individual PDF success
- `bibliography.availableInZotero`: Tooltip text

**Files updated**:
- `public/locales/en/common.json`
- `public/locales/fr/common.json`

---

## 📊 Implementation Statistics

### Summary
- **Total tasks in original plan**: 19 tasks
- **Tasks implemented**: 11 major features
- **Implementation rate**: 58%
- **Lines of code added**: ~800 LOC
- **Files modified**: 12 files
- **New components**: 1 (PDFSelectionDialog)

### What Was Implemented
✅ Phase 1: UI improvements (3 tasks)
✅ Phase 2: Zotero PDF detection (2 tasks)
✅ Phase 3: PDF download & indexation (4 tasks)
✅ Phase 4: Configuration improvements (2 tasks)

### What Was Postponed
See [BETA2_NOT_IMPLEMENTED.md](https://github.com/cliodeck/cliodeck-app/blob/beta2/BETA2_NOT_IMPLEMENTED.md) for detailed list of 8 postponed features.

**Key postponed features:**
- Bibliography update/sync function
- Modified PDF detection and re-indexation
- Orphan PDF detection and cleanup
- Merge Bibliography and PDFs tabs
- Statistics dashboard
- Multiple embedding model options
- Standalone embeddings without Ollama

**Why postponed**: Focus on core workflow quality over advanced features. These will be considered for BETA 3 and BETA 4.

---

## 🐛 Bug Fixes

### Topic Modeling Status
- **Issue**: Status showed "Unknown" instead of proper installation state
- **Fix**: Corrected state checking logic
- **Impact**: Users can now see if BERTopic is properly installed

### Theme Persistence
- **Issue**: Dense theme option was confusing and not well integrated
- **Fix**: Removed dense theme, improved light/dark themes spacing
- **Impact**: Cleaner UI, better readability

### Batch Indexing Progress
- **Issue**: Progress UI didn't show which citation was being processed
- **Fix**: Added current citation title to progress display
- **Impact**: Better visibility during long batch operations

---

## 📚 Documentation

### New Documentation
- **[Zotero Integration Guide](./1.5-Zotero-Integration-Guide.md)**: Complete guide for using Zotero features
  - Configuration steps
  - PDF detection and download
  - Batch operations
  - Troubleshooting
  - Best practices

### Updated Documentation
- **Installation Guide**: Updated prerequisites (Zotero optional but recommended)
- **Technical Architecture**: Added Zotero service documentation

---

## ⚠️ Breaking Changes

**None**. BETA 2 is fully backward compatible with BETA 1 projects.

However, note:
- **Theme setting migration**: If you were using "dense" theme, it will default to "auto" (light/dark based on time)
- **Configuration warnings**: You'll now see prominent warnings when changing embedding models or chunking config

---

## 🚀 Upgrade Instructions

### From BETA 1

1. **Backup your projects** (recommended)
2. **Install BETA 2** using your platform's package manager
3. **Open existing project** - no migration needed
4. **Configure Zotero** (if not already done):
   - Settings → Zotero Configuration
   - Enter User ID and API Key
   - Test connection
5. **Re-import bibliography** (optional):
   - Use merge mode to detect Zotero PDFs
   - Download missing PDFs with new batch button

### From Source

```bash
git checkout beta2
npm install
npx electron-rebuild -f
npm run build
npm start
```

---

## 🎓 Usage Tips

### Getting Started with BETA 2

**1. First-time Zotero setup:**
```
Settings → Zotero → Enter credentials → Test connection
```

**2. Import and download everything:**
```
Bibliography → Zotero Import → Select collection → Sync
Bibliography → Click 📥 Download All → Wait for completion
```

**3. Start writing with RAG:**
```
Open editor → Switch to RAG mode → Ask questions about your PDFs
```

### Best Practices

**For large libraries (100+ citations):**
- Import specific collections rather than entire library
- Download PDFs in batches of 50-100
- Use "Index All PDFs" after downloading locally
- Monitor disk space

**For active research:**
- Use merge mode when updating bibliography
- Download PDFs incrementally as you need them
- Re-index if you change chunking configuration
- Keep Zotero in sync with ClioDeck

**For optimal performance:**
- Configure RAG settings (embedding model, chunking) BEFORE batch indexing
- Use recommended embedding models (see warnings)
- Close other heavy applications during batch operations
- Consider CPU-optimized chunking for faster processing

---

## 🔮 Future Plans

### BETA 3 Candidates
Based on user feedback and development priorities:

1. **Bibliography Update/Sync**: Automatic detection of changes in Zotero
2. **Statistics Dashboard**: Citation counts, indexing status, storage usage
3. **Orphan PDF Detection**: Find and clean up unused PDFs
4. **Modified PDF Detection**: Re-index when Zotero PDFs are updated

### BETA 4 and Beyond
- Merge Bibliography and PDFs panels
- Multiple embedding model options
- Citation editing in ClioDeck
- Export annotated bibliography

See [BETA2_NOT_IMPLEMENTED.md](https://github.com/cliodeck/cliodeck-app/blob/beta2/BETA2_NOT_IMPLEMENTED.md) for complete list.

---

## 🙏 Acknowledgments

BETA 2 development was AI-assisted using Claude (Anthropic). All design decisions, testing, and quality assurance by the ClioDeck team.

**Contributors:**
- Frédéric Clavert (project lead)
- Claude AI (development assistance)

---

## 📝 Known Issues

### Limitations
- **No citation editing**: Citations must be edited in Zotero, then re-imported
- **No automatic sync**: Changes in Zotero require manual re-import
- **No PDF conflict resolution**: If PDF exists locally and in Zotero, manual intervention needed
- **Large PDFs**: Files over 50 MB may timeout during download

### Workarounds
- **Editing citations**: Edit in Zotero → Export BibTeX → Re-import with merge mode
- **Syncing updates**: Periodically re-sync collection with merge mode
- **PDF conflicts**: Delete local PDF first, then download from Zotero
- **Large PDFs**: Download manually from Zotero, add file path to citation

---

## 📞 Support

### Getting Help
- **Installation issues**: See [Installation Guide](./1.-ClioDeck-Installation.md)
- **Zotero issues**: See [Zotero Integration Guide](./1.5-Zotero-Integration-Guide.md)
- **Bug reports**: [GitHub Issues](https://github.com/cliodeck/cliodeck-app/issues)
- **Feature requests**: [GitHub Discussions](https://github.com/cliodeck/cliodeck-app/discussions)

### Reporting Bugs
When reporting issues, please include:
1. ClioDeck version (Help → About)
2. Operating system and version
3. Steps to reproduce
4. Expected vs. actual behavior
5. Console logs (View → Developer → Developer Tools → Console)

---

## 📄 License

ClioDeck is open source software licensed under [LICENSE](https://github.com/cliodeck/cliodeck-app/blob/main/LICENSE).

---

**Thank you for using ClioDeck BETA 2!**

Your feedback is essential for improving the tool. Please share your experience on GitHub.