Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ ALLOWED_HOSTS=localhost,127.0.0.1
CSRF_TRUSTED_ORIGINS=http://localhost,http://127.0.0.1,http://localhost:8080,http://127.0.0.1:8080

# Docker Compose runtime defaults. Host-side lint and test use .env.test instead.
DATABASE_URL=postgresql://newsletter:newsletter@postgres:5432/newsletter_maker
DATABASE_URL=postgresql://newsletter:newsletter@postgres:5432/digest_engine
REDIS_URL=redis://redis:6379/0
QDRANT_URL=http://qdrant:6333

OPENROUTER_API_KEY=
OPENROUTER_API_BASE=https://openrouter.ai/api/v1
OPENROUTER_APP_URL=
OPENROUTER_APP_NAME=newsletter-maker
OPENROUTER_APP_NAME=digest-engine

AI_CLASSIFICATION_MODEL=meta-llama/llama-3.1-70b-instruct
AI_RELEVANCE_MODEL=qwen/qwen-2.5-72b-instruct
Expand All @@ -32,7 +32,7 @@ OLLAMA_URL=http://ollama:11434

REDDIT_CLIENT_ID=
REDDIT_CLIENT_SECRET=
REDDIT_USER_AGENT=newsletter-maker/0.1
REDDIT_USER_AGENT=digest-engine/0.1

# Used to encrypt project-scoped Bluesky app passwords stored in the database.
# Set this to a stable secret in each environment.
Expand Down
4 changes: 2 additions & 2 deletions .env.test
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
DATABASE_URL=sqlite:///:memory:
OPENROUTER_API_KEY=test-key
OPENROUTER_API_BASE=https://openrouter.ai/api/v1
OPENROUTER_APP_NAME=newsletter-maker
OPENROUTER_APP_NAME=digest-engine
OLLAMA_URL=http://ollama:11434
REDDIT_CLIENT_ID=client
REDDIT_CLIENT_SECRET=secret
REDDIT_USER_AGENT=newsletter-maker/test
REDDIT_USER_AGENT=digest-engine/test
CELERY_BROKER_URL=memory://
CELERY_RESULT_BACKEND=cache+memory://
ALLOWED_HOSTS=localhost,127.0.0.1,nginx,testserver
Expand Down
6 changes: 3 additions & 3 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Newsletter Maker Project Instructions
# Digest Engine Project Instructions

You are working in Newsletter Maker, a Django + DRF + Celery + Qdrant backend with a Next.js App Router frontend.
You are working in Digest Engine, a Django + DRF + Celery + Qdrant backend with a Next.js App Router frontend.

## Repository Shape

- Backend runtime code is split across `core/`, `projects/`, `content/`, `entities/`, `ingestion/`, `newsletters/`, `pipeline/`, `trends/`, and `users/`.
- Django project settings and top-level URLs live in `newsletter_maker/`.
- Django project settings and top-level URLs live in `digest_engine/`.
- Backend tests live in app-local `tests/` packages first (`users/tests/`, `projects/tests/`, `ingestion/tests/`, `newsletters/tests/`, `pipeline/tests/`), with `core/tests/` reserved for the remaining cross-cutting coverage.
- The repo-root `tests/` package is for integration coverage only. New unit and app-scoped tests should live in the owning app's `tests/` package.
- Frontend application code lives in `frontend/src/app/`, shared UI in `frontend/src/components/`, and shared API/types/helpers in `frontend/src/lib/`.
Expand Down
4 changes: 2 additions & 2 deletions .github/instructions/backend-python.instructions.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
name: "Backend Python Guidelines"
description: "Use when editing Django, DRF, Celery, plugin, management command, or backend test code in Python. Covers project scoping, workflow placement, docstrings, and focused validation for core/, newsletter_maker/, tests/, and manage.py."
description: "Use when editing Django, DRF, Celery, plugin, management command, or backend test code in Python. Covers project scoping, workflow placement, docstrings, and focused validation for core/, digest_engine/, tests/, and manage.py."
applyTo:
- "core/**/*.py"
- "newsletter_maker/**/*.py"
- "digest_engine/**/*.py"
- "tests/**/*.py"
- "manage.py"
---
Expand Down
2 changes: 1 addition & 1 deletion .github/instructions/documentation.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ applyTo:
- When a behavior changes, update the closest existing document instead of adding a new overlapping explanation.
- Keep `docs/DEVELOPER_GUIDE.md` current when the best "where to look first" path changes for contributors.
- Keep `README.md` high-level. Put detailed runtime, workflow, or operator guidance in `docs/` and link to it.
- When documenting backend behavior, align the wording with the real implementation in files like `core/models.py`, `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/api.py`, and `newsletter_maker/settings/`.
- When documenting backend behavior, align the wording with the real implementation in files like `core/models.py`, `core/tasks.py`, `core/pipeline.py`, `core/newsletters.py`, `core/api.py`, and `digest_engine/settings/`.
- When documenting frontend behavior, align the wording with the real implementation in `frontend/src/app/`, `frontend/src/components/`, and `frontend/src/lib/`.
- If a doc mentions commands, prefer the repo's real commands from `justfile`, `package.json`, or `manage.py`.
- If a code change affects logging, relevance scoring, ingestion, newsletter intake, or onboarding, check whether `docs/LOGGING.md`, `docs/RELEVANCE_SCORING.md`, `docs/IMPLEMENTATION_OVERVIEW.md`, or `docs/DEVELOPER_GUIDE.md` should change too.
Expand Down
28 changes: 14 additions & 14 deletions .github/workflows/build-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,17 @@ jobs:
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4

- name: Lint chart
run: helm lint deploy/helm/newsletter-maker
run: helm lint deploy/helm/digest-engine

- name: Render chart
run: helm template newsletter-maker deploy/helm/newsletter-maker -f
deploy/helm/newsletter-maker/values-minikube.yaml >
/tmp/newsletter-maker-chart.yaml
run: helm template digest-engine deploy/helm/digest-engine -f
deploy/helm/digest-engine/values-minikube.yaml >
/tmp/digest-engine-chart.yaml

- name: Render staging overlay
run: helm template newsletter-maker-staging deploy/helm/newsletter-maker -f
deploy/helm/newsletter-maker/values-staging.yaml >
/tmp/newsletter-maker-staging-chart.yaml
run: helm template digest-engine-staging deploy/helm/digest-engine -f
deploy/helm/digest-engine/values-staging.yaml >
/tmp/digest-engine-staging-chart.yaml

build-frontend:
name: Build frontend
Expand Down Expand Up @@ -88,13 +88,13 @@ jobs:
- name: Build backend image
env:
DOCKER_BUILDKIT: "1"
run: docker build -t newsletter-maker-ci:${{ github.sha }} -f
run: docker build -t digest-engine-ci:${{ github.sha }} -f
docker/web/Dockerfile .

- name: Scan backend image with Trivy
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
image-ref: newsletter-maker-ci:${{ github.sha }}
image-ref: digest-engine-ci:${{ github.sha }}
scan-type: image
severity: HIGH,CRITICAL
ignore-unfixed: true
Expand All @@ -111,22 +111,22 @@ jobs:
- name: Publish backend image
if: github.event_name == 'push'
env:
IMAGE_REPOSITORY: ghcr.io/${{ github.repository_owner }}/newsletter-maker
IMAGE_REPOSITORY: ghcr.io/${{ github.repository_owner }}/digest-engine
run: |
set -euo pipefail

docker tag newsletter-maker-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:${GITHUB_SHA}
docker tag digest-engine-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:${GITHUB_SHA}
docker push ${IMAGE_REPOSITORY}:${GITHUB_SHA}

if [[ "${GITHUB_REF}" == "refs/heads/main" ]]; then
docker tag newsletter-maker-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:main
docker tag digest-engine-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:main
docker push ${IMAGE_REPOSITORY}:main
fi

if [[ "${GITHUB_REF}" == refs/tags/* ]]; then
version_tag="${GITHUB_REF#refs/tags/}"
docker tag newsletter-maker-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:${version_tag}
docker tag digest-engine-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:${version_tag}
docker push ${IMAGE_REPOSITORY}:${version_tag}
docker tag newsletter-maker-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:latest
docker tag digest-engine-ci:${GITHUB_SHA} ${IMAGE_REPOSITORY}:latest
docker push ${IMAGE_REPOSITORY}:latest
fi
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Newsletter Maker
# Digest Engine

![Image of AI-powered newsletter workflow](readme.jpg)

An AI-powered content curation platform for technically-oriented newsletters. Newsletter Maker ingests content from dozens of sources, builds authority models of people and companies in a domain, and surfaces the most relevant articles, trends, and themes for each edition — so editors spend their time writing, not searching.
An AI-powered content curation platform for technically-oriented newsletters. Digest Engine ingests content from dozens of sources, builds authority models of people and companies in a domain, and surfaces the most relevant articles, trends, and themes for each edition — so editors spend their time writing, not searching.

The system is organized into projects: each newsletter project has its own tracked entities, relevance model, and content pipeline. Projects are assigned to Django groups so editorial access can be shared cleanly. Designed for non-technical editors who don't know what a vector database is and don't need to.

Expand Down Expand Up @@ -68,7 +68,7 @@ just k8s-build-minikube
just k8s-install-minikube

# Terminal 2
kubectl port-forward svc/newsletter-maker-newsletter-maker-nginx 8080:80
kubectl port-forward svc/digest-engine-digest-engine-nginx 8080:80
```

- Admin UI: <http://localhost:8080/admin/>
Expand Down Expand Up @@ -99,7 +99,7 @@ For full workflows and troubleshooting, see [docs/developer-guide/local-developm

## What This Does That Existing Tools Don't

Tools like Feedly, UpContent, and ContentStudio handle parts of the content curation problem. Newsletter Maker combines several capabilities none of them offer:
Tools like Feedly, UpContent, and ContentStudio handle parts of the content curation problem. Digest Engine combines several capabilities none of them offer:

- **Authority scoring from newsletter cross-referencing.** By ingesting peer newsletters, the system builds an authority model based on who real editors actually link to — a human-curated endorsement signal no existing tool provides.
- **Per-project relevance training.** Upvote/downvote feedback trains a personalized relevance model per project. The tool learns what each editorial project considers valuable.
Expand Down Expand Up @@ -172,7 +172,7 @@ The system is designed for graceful failure, not silent corruption. Unparseable

## Project Documentation

Newsletter Maker documentation is organized by audience inside the `docs/` folder:
Digest Engine documentation is organized by audience inside the `docs/` folder:

- [User Guide](docs/user-guide/getting-started-saas.md) covers managing projects, intaking content, and curating drafts.
- [Admin Guide](docs/admin-guide/overview.md) covers installation, configuration, user management, and operational health.
Expand Down
13 changes: 7 additions & 6 deletions content/admin.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
"""Admin configuration for content-domain models."""

from typing import cast

from django.contrib import admin, messages
from django.db.models import Avg
from django.utils.html import format_html
from unfold.admin import ModelAdmin

from content.models import Content, UserFeedback
from core.settings_types import CoreSettings


def _score_to_percent(value):
Expand Down Expand Up @@ -124,6 +127,8 @@ def view_trace(self, obj):
from django.conf import settings
from django.urls import reverse

typed_settings = cast(CoreSettings, settings)

latest_skill_result = (
obj.skill_results.filter(superseded_by__isnull=True)
.order_by("-created_at")
Expand Down Expand Up @@ -173,12 +178,8 @@ def view_trace(self, obj):
trace_id = value
break

if (
not trace_url
and trace_id
and getattr(settings, "AI_TRACE_URL_TEMPLATE", "")
):
trace_url = settings.AI_TRACE_URL_TEMPLATE.format(
if not trace_url and trace_id and typed_settings.AI_TRACE_URL_TEMPLATE:
trace_url = typed_settings.AI_TRACE_URL_TEMPLATE.format(
content_id=obj.id,
run_id=trace_id,
skill_name=latest_skill_result.skill_name,
Expand Down
4 changes: 3 additions & 1 deletion content/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from drf_spectacular.utils import extend_schema
from rest_framework import serializers, status, viewsets
from rest_framework.decorators import action
from rest_framework.permissions import BasePermission
from rest_framework.response import Response

from content.models import Content, UserFeedback
Expand All @@ -12,8 +13,8 @@
CONTENT_CREATE_REQUEST_EXAMPLE,
CONTENT_RESPONSE_EXAMPLE,
PROJECT_ID_PARAMETER,
ProjectOwnedQuerysetMixin,
SKILL_NAME_PARAMETER,
ProjectOwnedQuerysetMixin,
build_crud_action_overrides,
document_project_owned_viewset,
)
Expand Down Expand Up @@ -54,6 +55,7 @@ class ContentViewSet(ProjectOwnedQuerysetMixin, viewsets.ModelViewSet):
def get_permissions(self):
"""Allow all members to read content, contributors to edit, and admins to delete."""

permission_classes: list[type[BasePermission]]
if self.action == "destroy":
permission_classes = [IsProjectAdmin]
elif self.action in {"create", "update", "partial_update", "run_skill"}:
Expand Down
8 changes: 4 additions & 4 deletions content/tests/test_admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ def test_high_value_filter_only_returns_high_value_reference_content(
model=Content,
model_admin=ContentAdmin(Content, AdminSite()),
)
filter_instance.value = lambda: "high_value"
cast(Any, filter_instance).value = lambda: "high_value"

filtered = filter_instance.queryset(_request(), Content.objects.all())

Expand Down Expand Up @@ -368,7 +368,7 @@ def test_duplicate_state_filter_returns_canonical_rows_with_duplicate_signals(
model=Content,
model_admin=ContentAdmin(Content, AdminSite()),
)
filter_instance.value = lambda: "canonical_with_duplicates"
cast(Any, filter_instance).value = lambda: "canonical_with_duplicates"

filtered = filter_instance.queryset(_request(), Content.objects.all())

Expand Down Expand Up @@ -406,7 +406,7 @@ def test_duplicate_state_filter_returns_suppressed_duplicates(
model=Content,
model_admin=ContentAdmin(Content, AdminSite()),
)
filter_instance.value = lambda: "suppressed_duplicates"
cast(Any, filter_instance).value = lambda: "suppressed_duplicates"

filtered = filter_instance.queryset(_request(), Content.objects.all())

Expand Down Expand Up @@ -558,7 +558,7 @@ def test_high_value_filter_lookups_and_noop_queryset(source_admin_context):
model=Content,
model_admin=ContentAdmin(Content, AdminSite()),
)
filter_instance.value = lambda: None
cast(Any, filter_instance).value = lambda: None
content = Content.objects.create(
project=source_admin_context.project,
url="https://example.com/high-value-noop",
Expand Down
1 change: 0 additions & 1 deletion content/tests/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
from content.models import Content, FeedbackType, UserFeedback
from entities.models import Entity
from pipeline.models import SkillResult, SkillStatus
from projects.model_support import SourcePluginName
from projects.models import Project, ProjectMembership, ProjectRole
from trends.models import ThemeSuggestion, ThemeSuggestionStatus, TopicCluster

Expand Down
16 changes: 8 additions & 8 deletions core/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@
search_similar_content,
)
from core.llm import build_skill_user_prompt, get_skill_definition, openrouter_chat_json
from digest_engine.telemetry import trace_span
from entities.extraction import run_entity_extraction
from entities.models import EntityMention
from newsletter_maker.telemetry import trace_span
from pipeline.models import (
ReviewQueue,
ReviewReason,
Expand Down Expand Up @@ -175,13 +175,13 @@ def get_ingestion_graph():
"""

graph = StateGraph(PipelineState)
graph.add_node("deduplicate", deduplicate_node)
graph.add_node("classify", classify_node)
graph.add_node("extract_entities", extract_entities_node)
graph.add_node("score_relevance", relevance_node)
graph.add_node("summarize", summarize_node)
graph.add_node("archive", archive_node)
graph.add_node("queue_review", queue_review_node)
graph.add_node("deduplicate", cast(Any, deduplicate_node))
graph.add_node("classify", cast(Any, classify_node))
graph.add_node("extract_entities", cast(Any, extract_entities_node))
graph.add_node("score_relevance", cast(Any, relevance_node))
graph.add_node("summarize", cast(Any, summarize_node))
graph.add_node("archive", cast(Any, archive_node))
graph.add_node("queue_review", cast(Any, queue_review_node))
graph.set_entry_point("deduplicate")
graph.add_conditional_edges(
"deduplicate",
Expand Down
1 change: 1 addition & 0 deletions core/settings_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@


class CoreSettings(Protocol):
AI_TRACE_URL_TEMPLATE: str
BLUESKY_CREDENTIALS_ENCRYPTION_KEY: str
CELERY_TASK_ALWAYS_EAGER: bool
DEFAULT_FROM_EMAIL: str
Expand Down
10 changes: 5 additions & 5 deletions core/tests/test_entrypoints.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from channels.routing import ProtocolTypeRouter

from newsletter_maker.celery import app
from digest_engine.celery import app


def _import_fresh(module_name: str):
Expand All @@ -18,10 +18,10 @@ def test_asgi_module_sets_default_settings_and_builds_application(mocker):
"django.core.asgi.get_asgi_application", return_value="asgi-app"
)

module = _import_fresh("newsletter_maker.asgi")
module = _import_fresh("digest_engine.asgi")

setdefault_mock.assert_called_once_with(
"DJANGO_SETTINGS_MODULE", "newsletter_maker.settings"
"DJANGO_SETTINGS_MODULE", "digest_engine.settings"
)
get_app_mock.assert_called_once_with()
assert module.django_asgi_application == "asgi-app"
Expand All @@ -36,10 +36,10 @@ def test_wsgi_module_sets_default_settings_and_builds_application(mocker):
"django.core.wsgi.get_wsgi_application", return_value="wsgi-app"
)

module = _import_fresh("newsletter_maker.wsgi")
module = _import_fresh("digest_engine.wsgi")

setdefault_mock.assert_called_once_with(
"DJANGO_SETTINGS_MODULE", "newsletter_maker.settings"
"DJANGO_SETTINGS_MODULE", "digest_engine.settings"
)
get_app_mock.assert_called_once_with()
assert module.application == "wsgi-app"
Expand Down
8 changes: 4 additions & 4 deletions core/tests/test_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ def test_openrouter_chat_json_requires_api_key(settings):
def test_openrouter_chat_json_posts_expected_request(settings, mocker):
settings.OPENROUTER_API_KEY = "test-key"
settings.OPENROUTER_API_BASE = "https://openrouter.example/api/v1/"
settings.OPENROUTER_APP_URL = "https://newsletter-maker.example"
settings.OPENROUTER_APP_NAME = "newsletter-maker"
settings.OPENROUTER_APP_URL = "https://digest-engine.example"
settings.OPENROUTER_APP_NAME = "digest-engine"
settings.AI_REQUEST_TIMEOUT_SECONDS = 12.5

response = SimpleNamespace(
Expand All @@ -44,8 +44,8 @@ def test_openrouter_chat_json_posts_expected_request(settings, mocker):
assert post_mock.call_args.kwargs["headers"] == {
"Authorization": "Bearer test-key",
"Content-Type": "application/json",
"HTTP-Referer": "https://newsletter-maker.example",
"X-OpenRouter-Title": "newsletter-maker",
"HTTP-Referer": "https://digest-engine.example",
"X-OpenRouter-Title": "digest-engine",
}
assert post_mock.call_args.kwargs["json"] == {
"model": "openrouter/test-model",
Expand Down
Loading
Loading