Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 7 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ The server component written in Go. This handles data synchronization, storage,
### **synkronus-cli**
A command-line utility to interact with the Synkronus server. Use it to manage custom app data, handle user administration, export data to Parquet format, and perform various administrative tasks.

### **synkronus-portal**
A web-based version of the the **synkronus-cli**

## We're Young & Fresh! 🌱🌱🌱

ODE is a **young and vibrant open-source project**, and we're incredibly welcoming to contributors of all experience levels and interests! Whether you're passionate about:
Expand All @@ -47,43 +50,11 @@ We believe that diverse perspectives and varied skill sets make our project stro
- Share how you're using ODE in your work

## CI/CD Pipeline
This monorepo uses GitHub Actions for CI/CD. For details (trigger conditions, tags, and workflows), see `.github/CICD.md`.

This monorepo uses GitHub Actions for continuous integration and deployment:

### Current Pipelines

#### Synkronus Docker Build & Publish
- **Trigger**: Push to `main` branch or pull requests affecting `synkronus/` directory
- **Registry**: GitHub Container Registry (ghcr.io)
- **Image**: `ghcr.io/opendataensemble/synkronus`
- **Tagging Strategy**:
- `main` branch → `latest` + `v{version}` (release versions)
- Other branches → `{branch-name}` (pre-release versions)
- **Workflow**: `.github/workflows/synkronus-docker.yml`

### Image Versioning

Images follow semantic versioning:
- **Release**: `ghcr.io/opendataensemble/synkronus:latest` or `ghcr.io/opendataensemble/synkronus:v1.0.0`
- **Pre-release**: `ghcr.io/opendataensemble/synkronus:develop` or `ghcr.io/opendataensemble/synkronus:feature-xyz`

### Using Published Images

Pull and run the latest Synkronus image:

```bash
docker pull ghcr.io/opendataensemble/synkronus:latest
docker run -d -p 8080:8080 \
-e DB_CONNECTION="postgres://user:pass@host:5432/synkronus" \
-e JWT_SECRET="your-secret-key" \
-v synkronus-bundles:/app/data/app-bundles \
ghcr.io/opendataensemble/synkronus:latest
```

**Documentation:**
- [CI/CD Pipeline Details](.github/CICD.md) - Comprehensive CI/CD documentation
- [Synkronus Docker Guide](synkronus/DOCKER.md) - Quick start guide
- [Synkronus Deployment Guide](synkronus/DEPLOYMENT.md) - Production deployment
- Synkronus Docker build & publish: `.github/workflows/synkronus-docker.yml`
- Formulus Android build (includes Formplayer asset build): `.github/workflows/formulus-android.yml`
- Synkronus deployment docs: `synkronus/DOCKER.md`, `synkronus/DEPLOYMENT.md`

## Code Quality: Linting & Formatting

Expand Down
18 changes: 8 additions & 10 deletions packages/components/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A comprehensive component library for the Open Data Ensemble (ODE) ecosystem. Th
- **Modern Minimalist Design** - Clean, simple, and aesthetically beautiful components
- **Platform-Specific** - Optimized implementations for React Native and React Web
- **Token-Based** - Built on `@ode/tokens` for consistent design
- **Accessible** - WCAG compliant with proper ARIA attributes
- **Accessible** - supports `accessibilityLabel` (web: `aria-label`, React Native: native accessibility props where supported)
- **Responsive** - Works seamlessly across all device sizes
- **Dark Mode Ready** - Supports light and dark themes

Expand Down Expand Up @@ -91,13 +91,12 @@ function App() {

### Button

Modern minimalist button with a unique fading border effect. When two buttons are placed together in a `ButtonGroup`, they automatically have opposite styles.
Modern minimalist button with transparent/default styling and token-based colors. When used inside a `ButtonGroup`, paired buttons automatically use contrasting variants for visual emphasis.

**Features:**
- Fading border effect on one end (left button fades on right, right button fades on left)
- Transparent background with border-colored text by default
- On hover: background fills with border color, text changes to contrast color
- Automatic opposite styling when paired
- On hover/active: background fills with the border color, and text switches to a contrasting color
- Automatic contrasting styling when paired in a `ButtonGroup`

```tsx
// Single button
Expand All @@ -113,7 +112,7 @@ Modern minimalist button with a unique fading border effect. When two buttons ar
```

**Props:**
- `variant`: `'primary' | 'secondary' | 'neutral'` (default: `'primary'`)
- `variant`: `'primary' | 'secondary' | 'neutral' | 'danger'` (default: `'primary'`)
- `size`: `'small' | 'medium' | 'large'` (default: `'medium'`)
- `disabled`: `boolean` (default: `false`)
- `loading`: `boolean` (default: `false`)
Expand Down Expand Up @@ -185,7 +184,7 @@ All components follow these design principles:

1. **Minimalism** - Clean, uncluttered interfaces that enhance usability
2. **Consistency** - Unified design language using ODE tokens
3. **Accessibility** - WCAG compliant with proper contrast ratios and touch targets
3. **Accessibility** - supports `accessibilityLabel`/ARIA where applicable
4. **Responsiveness** - Adapts seamlessly to all screen sizes
5. **Performance** - Lightweight and optimized for fast rendering

Expand All @@ -194,9 +193,8 @@ All components follow these design principles:
The button component features a unique design inspired by modern minimalist aesthetics:

- **Default State**: Transparent background with colored border and matching text
- **Fading Border**: Border fades on one end (left button fades right, right button fades left)
- **Hover State**: Background fills with border color, text changes to high-contrast color
- **Paired Buttons**: When two buttons are together, they have opposite color schemes
- **Hover State**: Background fills with border color, text changes to a high-contrast color
- **Paired Buttons**: When two buttons are together, they use contrasting color variants
- **Smooth Transitions**: All state changes use smooth cubic-bezier animations

## Platform Differences
Expand Down
159 changes: 3 additions & 156 deletions synkronus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,30 +38,7 @@ synkronus/

**For production deployment**, we recommend using Docker Compose with nginx and cloudflared tunnel.

See [DOCKER.md](./DOCKER.md) for quick start or [DEPLOYMENT.md](./DEPLOYMENT.md) for comprehensive deployment guide.

```bash
# Quick start with docker-compose
cp docker-compose.example.yml docker-compose.yml
# Edit docker-compose.yml with your secrets
docker compose up -d

# Or run directly with Docker
docker pull ghcr.io/opendataensemble/synkronus:latest
docker run -d -p 8080:8080 \
-e DB_CONNECTION="postgres://user:pass@host:5432/synkronus" \
-e JWT_SECRET="your-secret-key" \
-v synkronus-bundles:/app/data/app-bundles \
ghcr.io/opendataensemble/synkronus:latest
```

When using the provided `docker-compose` setup, the recommended way to initialize databases is:

- Run a single PostgreSQL container (service `postgres`).
- Use `docker compose exec postgres psql -U postgres` to create one or more databases and users for Synkronus.
- Point each Synkronus instance at its own database via the `DB_CONNECTION` string.

See [DEPLOYMENT.md](./DEPLOYMENT.md) for the full step-by-step database initialization procedure.
See [DOCKER.md](./DOCKER.md) for running pre-built images and local Docker usage, and [DEPLOYMENT.md](./DEPLOYMENT.md) for full production setup (including database initialization).

### Development Setup

Expand Down Expand Up @@ -122,145 +99,15 @@ go run cmd/synkronus/main.go

## Deployment Architecture

Synkronus is designed to be deployed as a single Docker container with multiple processes managed by a supervisor:

```
┌─────────────────────────────────────────┐
│ Docker Container │
│ │
│ ┌─────────┐ ┌───────────────┐ │
│ │ Nginx │───────▶│ Synkronus │ │
│ │ │ │ Go Server │ │
│ └─────────┘ └───────────────┘ │
│ │ │ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────┐ ┌───────────────┐ │
│ │ Static │ │ PostgreSQL │ │
│ │ UI │ │ Database │ │
│ └─────────┘ └───────────────┘ │
│ │
└─────────────────────────────────────────┘
```

### Benefits

- **Simple deployment**: Single container to manage
- **Proper separation of concerns**: Nginx handles TLS, static files, and proxying
- **Easy scaling**: Can be deployed to any container orchestration system

### Implementation Details

- **TLS/Let's Encrypt**: Automatic certificate management via Certbot with Nginx
- **UI Integration**: Static SPA (Single Page Application) served by Nginx
- **API Proxying**: Nginx proxies API requests to the Go server
- **Data Persistence**: PostgreSQL database for robust data storage
- **Configuration**: Environment variables for flexible deployment options
For Docker-based deployment details (pre-built images, docker-compose configuration, and production setup), see `DOCKER.md` and `DEPLOYMENT.md`.

## API Documentation

API documentation is generated from the OpenAPI specification in `openapi/synkronus.yaml`.

## Sync protocol

Attachments (e.g. photos, audio recordings) are **binary blobs** referenced by observations. They are stored and transferred separately from the observation metadata to simplify synchronization, improve offline support, and reduce conflicts.

### Core design principles

- **Immutable attachments**
- Once an attachment is uploaded, it cannot be modified in place.
- Any "update" requires creating a new attachment with a new unique ID (typically a GUID).

- **Separation of concerns**
- Observation records sync via `/sync/pull` and `/sync/push`.
- Attachments are uploaded and downloaded via dedicated endpoints.
- This design ensures simpler, smaller payloads and clearer transaction boundaries.

- **Stateless server**
- The server does not maintain per-client state about which attachments have been uploaded or downloaded.
- Clients manage their own attachment sync state.

### Conflict avoidance

- Because attachment IDs are generated as GUIDs client-side, filename clashes are extremely unlikely.
- The server can include a simple existence check to reject accidental overwrites.

### Clean-up and maintenance

- Server can run periodic jobs to:
- Identify orphaned attachments (no observation references).
- Prune old or unused files.
- Enforce retention policies.

### Security considerations

- Require authentication (e.g. bearer tokens) for all attachment endpoints.
- Validate file types and sizes on upload to prevent abuse.
- Optionally scan files for malware.

### Example implementation notes

- Filesystem storage for MVP:
- Simple to implement and inspect.
- Suitable for single-server deployments.

- Cloud object storage (e.g. S3, GCS) for future:
- Supports presigned URLs for direct client upload/download.
- Handles scalability and redundancy.
- Reduces server load.

### Advantages of this server design

- Keeps observation data sync API simple and JSON-only.
- Supports large binary payloads without bloating /sync/push or /sync/pull requests.
- Encourages incremental, resumable, offline-friendly sync strategies.
- Leaves attachment sync policy (e.g. lazy, bulk, partial) up to the client.
- Scales from MVP file-system storage to cloud-native solutions.

## Minimal Sync Protocol Design

This project uses a minimal sync approach for the `/sync/pull` and `/sync/push` endpoints that remains compatible with WatermelonDB's client-side helpers without requiring a full server-side change-tracking table.

### Approach

- Server stores a monotonic version number (via PostgreSQL trigger) for every observation change.
- Each Observation record includes `created_at`, `updated_at`, and `deleted` fields.
- Server simply returns all observations changed since the client's last known version.

### Client-side adaptation

- The client *infers* WatermelonDB's `_status`:
- `deleted` → `_status: "deleted"`
- `created_at == updated_at` → `_status: "created"`
- Else → `_status: "updated"`
- `_changed` is hardcoded to `"data,deleted"` since most content is stored in the `data` JSON column.
- The client wraps the server's flat `records` array into the expected WatermelonDB format:

```json
{
"changes": {
"observations": [ ... ]
},
"timestamp": current_version
}
```

### Advantages

- Requires no server-side schema changes or change-tracking tables.
- Enables immediate shipping.
- Compatible with WatermelonDB's `experimentalApplyRemoteChanges`.
- Easy to side-load data or perform manual database edits on the server.

### Trade-offs

- Client carries responsibility for inferring `_status` and `_changed`.
- Always sends/receives full `data` blobs.
- Less granular change tracking.

### Future Consideration

This minimal approach is ideal for the current single-table use case. As requirements grow (e.g. multiple tables or better conflict resolution), the server can evolve to maintain an explicit changes log and deliver a richer sync protocol with exact `_status` and `_changed` values. See [WatermelonDB/Sync](https://watermelondb.dev/docs/Sync/Intro) for more details on how to align more closely with the WatermelonDB approach to sync.
For the sync protocol design details (record model, attachment handling, pagination, and conflict strategy), see [`documentation/sync-protocol.md`](./documentation/sync-protocol.md).

## License

Expand Down
2 changes: 1 addition & 1 deletion synkronus/documentation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
- Optional API versioning via `x-api-version` header
- Optional ETag support for caching and efficiency

Full OpenAPI spec lives in [`Synkronus Openapi`](Synkronus/Openapi.yaml)
Full OpenAPI spec lives in [`synkronus/openapi/synkronus.yaml`](../openapi/synkronus.yaml)

---

Expand Down
Loading