Repository navigation
Releases
Nogon (Dev Build)
v0.21.0
New Features
Configuration Reload
hister reload applies configuration, rules, and logging changes without a
manual server restart. It waits for completion and rejects invalid settings.
Expect a brief interruption and refresh open web pages afterward. Client
configuration changes still require restarting the affected clients.
More Expressive Search
Combine nested groups, phrases, and field filters with AND, OR, and NOT.
For example, (site:github.com OR site:gitlab.com) NOT title:archived searches
both sites while excluding archived titles. The new site: filter includes
subdomains, and has: finds documents with a populated field or metadata value,
such as has:label or has:metadata.author.
Invalid queries now explain what needs fixing and where. URL regular expression
searches are faster when a pattern has a specific starting prefix.
Kiwix, Sitemap, and Readwise Reader Imports
hister import zim collection.zim makes HTML and plain text from Kiwix archives
searchable without downloading each original page. It recovers original URLs
and article links, with --base-url as a fallback when archive paths match the
source website. Entries without an identifiable original URL are skipped with
a warning. Embedded images and other assets are not imported, and large
archives can require substantial memory.
hister import sitemap FILE_OR_URL indexes pages listed in XML sitemaps,
including sitemap indexes, gzip compressed files, and standard input. Duplicate
URLs are queued once, and the page crawl can be interrupted and resumed using
its job ID. Existing documents are skipped unless --force is supplied.
hister import readwise imports Readwise Reader documents with saved content,
notes, summaries, tags, and source metadata. It uses Reader's stored HTML when
available and otherwise tries the original page. Separate highlight and note
records are skipped, and original PDF and EPUB attachments are not downloaded.
Language Controls and Persistent Corrections
Limit automatic language detection with indexer.languages, or choose between
the default low memory mode and high accuracy mode with
indexer.language_detection_accuracy. Short web pages can use their titles to
help identify the language, and matching page declarations can supply regional
labels.
hister update QUERY --language en now pins the language across reindexing,
imports, and document updates. Use --language auto to clear a pin or
--language unknown to keep a document without an identified language.
language:en includes regional variants such as en-US, while a regional filter
narrows the results. Existing documents remain searchable when automatic
detection is disabled.
Expanded MCP Tools for AI Assistants
Assistants can page through search results, use the new get_facets tool to
explore document counts, and filter history by text and date. Search limits now
apply to the combined history, keyword, and semantic results.
Preview requests can select text, HTML, or metadata and read long documents in
smaller portions, reducing how much content an assistant needs to retrieve at
once.
Remote User Management and Proxy Authentication
The new hister user create, show, update, and delete commands manage
accounts on the configured running server using an administrator's token.
Account changes are applied together, conflicting updates are rejected, and
the last administrator cannot be deleted or demoted. Use
hister user create USERNAME --offline --admin with the server stopped to create
the first administrator or recover access.
server.proxy_auth_header enables sign in through a trusted authentication
proxy. The proxy must verify users, replace client supplied identity headers,
and prevent direct access to Hister. New proxy users receive ordinary accounts
without passwords. OIDC providers can also select a stable account identity
with identity_claim, including providers that do not supply email addresses.
Enhancements
- Search interface: simpler result controls, clearer domains and titles,
compact expandable snippets, and mobile friendly actions make results easier
to scan. Setapp.snippet_lengthto control the returned context and use
Always expand snippets to remember a browser preference. Suggestions are
hidden by default and can be enabled through Actions → Show suggestions. - Semantic search responsiveness: query embedding requests have separate
capacity from indexing and a configurable timeout. Searches return keyword
results when query embedding times out, with a default limit of two seconds. - PDF extraction: the built in reader handles more LaTeX generated PDFs.
Configureindexer.pdf_parserto use an external parser such aspdftotext
for browser uploads, watched files, and file imports. - Browser capture: the extension uses canonical URLs on the same hostname
to reduce duplicate pages. Automatic capture skips HTTP error pages, and
PDFs are recognized by their response type even without a.pdfsuffix. - Previews and login: GitHub issue previews include discussion content,
failed previews offer a fallback, and signing in returns to the original
page with its query or preview link. - Feedback: maintenance commands show progress, command line searches
explain empty results and can include document text, and the Add page
explains submission failures. - Monitoring:
server.expose_versionenables public version reporting for
monitoring clients and is disabled by default. Prometheus latency metrics
use native histograms. - Dependencies: Go modules, npm packages, container bases, Nix inputs, and
GitHub Actions were updated.
Bug Fixes
- Anonymous searches in public mode return only global documents. Submitted
documents cannot reference another document's stored HTML or favicon, and
client redirects no longer leak access tokens to other destinations. - Password changes invalidate the affected user's existing browser sessions.
GitHub login uses permanent account IDs so renamed or recycled usernames
cannot change which identity an account belongs to. - Persistent crawls mark URLs complete only after successful document
submission. Failed submissions are recorded as failures, and interrupted
submissions remain resumable. - Cleanup preserves HTML and favicons while documents are still being indexed.
Reindexing keeps files whose extracted text is empty, and exports preserve
existing backups when writing fails. - Queries containing only negated terms return matching documents. Relevance
optimization no longer broadens the requested results, and semantic results
honor date filters and requested content fields. - Server access to local files is restricted to configured directories,
including checks on symbolic link targets. Org files cannot read other files
through include directives. Request, download, and extraction limits bound
resource use. - PDF capture preserves pending downloads and uses the committed page title.
The Firefox crawler rejects reported HTTP error responses, and qutebrowser
handles larger browser events. - PostgreSQL timestamps are handled correctly, priority rule errors are
reported, and repeated index write failures no longer cause excessive CPU
use.
Backward Compatibility Notes
The following changes require attention when upgrading from v0.20.0:
- GitHub and OIDC accounts: existing GitHub account links are not
automatically migrated to permanent account IDs. Back up the database and
arrange verified account relinking before users sign in again; otherwise,
login creates a separate account while documents remain with the old one.
Changing OIDC'sidentity_claimalso requires relinking existing accounts.
Leaving it unset preserves the existing email based identity. - User management scripts: the old
create-user,show-user,
update-user, anddelete-usernames remain deprecated aliases, but account
operations now target the running server and require an administrator's
personal token. Useuser create --offlinefor local account creation with
the server stopped. Replace the old token display flag with--show-token;
--tokennow supplies authentication credentials. - Search syntax: uppercase
AND,OR, andNOTare operators. Quote them
to search for the words themselves. Negation takes precedence overAND,
thenOR; use parentheses to make grouping explicit. Malformed queries are
rejected by search, update, and delete operations. - Local files: run
hister reindexto remove previously indexed files
reached through symbolic links outside configured directories. Org files
using#+INCLUDEor#+SETUPFILEmust contain the needed text directly.
Opening a watched local file downloads it; use its preview to read it in
Hister. - Size limits: document, PDF, and batch uploads default to 40 MiB, controlled
byserver.max_batch_body_size. HTTP HTML downloads and extracted text have
a 32 MiB ceiling. Favicon and robots files have an 8 MiB limit; an oversized
robots file causes the site's URLs to be skipped. Raising the upload limit
does not raise extraction limits. - MCP clients: requests carrying an unrelated
Originare rejected. Set
server.base_urlto the public URL when using a reverse proxy. Clients
paging opened history should pass bothlast_idandlast_updated_atfrom
the previous response. - Language settings: reindex to apply detection changes to existing
documents. Reapply language corrections made before persistent pins were
available if they should survive future updates.
v0.20.0
New Features
Browser Bookmark Imports
hister import browser bookmarks imports bookmarks from Firefox based browsers,
Chromium based browsers, and Ladybird. It supports automatic detection and
explicit --browser and --db options, applies the bookmarks label by default,
and uses persistent crawl jobs that can be interrupted and resumed.
The new hister import browser history command accepts the same named options.
The existing positional history command and older browser import jobs remain
supported.
Raindrop.io Imports
hister import raindrop imports bookmarks through the Raindrop.io API or from a
CSV export with --input. API imports preserve titles, dates, notes, highlights,
annotations, tags, favorites, and collection paths. Linked pages are downloaded
with the configured crawler, while bookmark details are retained if a page
cannot be retrieved. CSV input can also be read from standard input.
Continuous File Imports
hister import file --watch performs an initial scan and keeps remote file
snapshots updated as files change. It supports explicit files, recursive
directories, or configured directory rules, including labels and ownership.
Temporary server failures are retried while the command runs.
Watch mode handles file snapshots rather than restoring exports or archives.
Removing or renaming a source file leaves its previously imported snapshot in
the index.
Configuration and Diagnostics
The new hister config commands create default configuration, report its path,
show effective settings with credentials redacted, and validate configuration
and extractor options. Inspection does not create runtime files.
hister doctor checks local configuration and extractor dependencies, then
verifies server connectivity, authentication, index compatibility, and server
extractor dependencies. It supports structured output and uses the new
GET /api/diagnostics endpoint, which requires an administrator in multiple
user mode.
Allow Rules and Guided Rule Editing
The new allow rule group restricts indexing to URLs matching at least one
allowed pattern. Skip rules still take precedence, and an empty allow list
preserves the existing behavior. The Rules page can now create patterns from
domains or exact URLs, optionally include subdomains, and validate and test
patterns before saving them.
Manual indexing from the browser extension and --ignore-rules on hister index
or import commands can override allow and skip rules. This choice is saved with
the document and survives reindexing. Sensitive content checks, robots rules,
and source selection filters still apply.
Structured Command Line Output
Search, indexing summaries, crawl inspection, diagnostics, and file and service
import summaries share text, JSON, JSONL, and CSV output formats. Search results
stream as pages arrive and preserve selected field order in text and CSV.
Indexing reports indexed, skipped, and failed counts and continues after
individual URL failures. hister index --failed-urls PATH writes failed URLs to
a file that can be reused with --input.
Prometheus Monitoring and Unix Sockets
Setting server.metrics: true enables GET /metrics with search, indexing,
document count, storage, Go runtime, and process metrics. The endpoint follows
the configured base path and authentication settings and requires an
administrator in multiple user mode, including when public mode is enabled.
The server can listen on a Unix socket using a server.address such as
unix:/run/hister/hister.sock. An explicit public server.base_url is required
for clients connecting through a reverse proxy. The socket is removed on a
clean shutdown.
Enhancements
- Browser extension: automatic capture reduces redundant submissions and
preserves final page snapshots during navigation and tab closure. - Search interface: loading feedback is clearer, filters and sort controls
remain available for empty results, and history results display index counts. - Web interface: the Add page clarifies file import and indexing options,
the profile page can copy access tokens, and the menu links to the browser's
extension store. - OAuth: GitHub, Google, and OIDC logins use S256 PKCE by default. GitHub
token exchanges now send credentials in a form POST. - Imports and backups: import failures provide clearer explanations,
backups receive a.jsonextension when omitted, and sensitive content
matches are easier to identify. - Offline commands: operations that only read SQLite data open the database
in read only mode. - Documentation: new monitoring and Homebrew instructions accompany expanded
crawler backend, import, configuration, and command line guides. - Dependencies: Go modules, npm packages, container bases, Nix inputs, and
GitHub Actions were updated.
Bug Fixes
- SQLite vector stores tolerate changed embedding dimensions at startup, and
reindexing recreates the store with the configured dimensions. A failed
vector store rebuild now aborts reindexing instead of silently proceeding. - Embedding retries recognize llama.cpp physical batch size errors.
- Semantic search respects the selected result order, removes deleted results
from the interface, and initially uses the server's configured defaults. - Languages without a registered Bleve analyzer use the default index instead
of attempting to create an unsupported language index. - Local file indexing applies configured labels, watches directory roots that
are symbolic links, and handles home directory and absolute paths correctly. - Imported HTML files follow the appropriate saved page or file snapshot path,
and the index command initializes configured extractors. - Database timestamps are stored consistently in UTC, with existing SQLite
timestamps normalized while preserving their instants and precision. - Browser import detection handles profile paths more reliably, including
Ladybird profiles, and retains support for resuming older history jobs. - Safari and other browser extensions receive valid CORS preflight responses,
including when custom authentication headers are used. - Text previews escape content when HTML is unavailable. Favicon responses use
stricter headers, and oversized thumbnails are rejected. - Lobsters extraction avoids duplicate comments, and Mastodon extraction no
longer applies incorrect link resolution. - History loading no longer blocks navigation, empty results clear stale
previews, and the TUI refreshes its viewport when a WebSocket connects. - Client errors preserve server explanations for HTTP 404 responses.
- FreeBSD builds work again, and the systemd service correctly excludes
privileged system calls and uses paths compatible with user services.
Backward Compatibility Notes
The following changes require attention when upgrading from v0.19.0:
- Command line automation:
--format jsonconsistently returns an array,
including single record summaries. Indexing, file imports, and service
imports return status2when processing completes with item failures and
status1for errors that prevent completion. Update scripts that depend on
earlier output shapes or exit statuses. - Configuration creation: use
hister config createin place of
hister create-config. The old command remains as a deprecated alias and
writes its notice to standard error. Configuration creation refuses to
overwrite an existing file. - OAuth providers: providers that reject PKCE parameters need
disable_pkce: truein their provider configuration. Restart logins begun
before the upgrade; existing authenticated sessions remain valid. Custom
GitHub token endpoints and proxies must accept form POST requests. - Allow rules and reindexing: existing rules without an
allowgroup retain
their behavior. Once allow rules are added, reindexing removes documents
excluded by allow or skip rules unless they carry an explicit manual
override. - SQLite timestamps: an automatic migration normalizes stored timestamps
to UTC. Command line displays continue to use local time.
v0.19.0
New Features
ChatGPT and Hacker News Extractors
New ChatGPT and Hacker News extractors preserve complete conversations and
discussion threads. ChatGPT extraction keeps visible user and assistant turns,
including headings, lists, tables, code blocks, and safe links. It supports
authenticated conversations, public shares, and custom GPT conversation URLs.
Hacker News extraction records submission metadata, optional post text, and the
complete nested comment tree. Shared discussion utilities also improve how
Reddit and Discourse comments are represented.
URL Regular Expression Search and Cleanup
Queries can now use url_re: to match normalized document URLs with Go regular
expressions. Matching supports anchors, case folding, negation, and grouped
alternatives. Invalid or excessively long expressions return a clear error.
The delete API supports a dry_run mode that counts matches without changing
the index. The Rules page and result actions can use this preview to delete
documents that match a new or edited skip rule. Hister shows the match count
and asks for confirmation while keeping the saved rule if deletion is canceled.
Safari History Imports
hister import browser can now import Safari history on macOS, including visit
counts and start date filtering. Browser databases are detected from their
schema instead of their filename, which also resolves ambiguous Safari and
Ladybird database names. Permission errors explain the Full Disk Access setting
required by macOS.
Terminal Focus and Native Appearance
The terminal interface now separates search input from result navigation.
Keyboard, pointer, and scroll actions move focus predictably, while contextual
shortcuts and feedback reflect the active area. Dialogs, tabs, rules, previews,
and narrow layouts have more consistent navigation and pointer behavior.
The new default terminal appearance inherits the terminal foreground,
background, and ANSI palette. The Settings overlay can switch among terminal,
automatic, dark, and light modes, while the theme picker remains available for
choosing complete Hister themes.
Richer Suggestions and Previews
OpenSearch suggestions now include pinned and history results before index
matches, together with descriptions and direct target URLs. Local and remote
file suggestions open the appropriate file or readable preview.
Readable previews display document type, language, label, visit count,
ownership, and extractor metadata. They also provide links to the source and a
standalone preview. Exact document identifiers keep previews and version
history on the correct document when users index the same URL separately.
Enhancements
- Home page: indexed pages, files, rules, and aliases use compact clickable
statistics, including a direct file search. - Web navigation: frequently used destinations are visible again, and the
menu provides automatic, light, and dark appearance choices. - User administration:
hister update-user --passwordcan assign or change
a password, including for an account originally created through OAuth. Token
regeneration now requires confirmation in the profile interface. - Semantic search: index metadata records the embedding configuration and
warns when stored vectors need reindexing. PostgreSQL configurations are
validated against the 2000 dimension limit of its HNSW vector index. - Indexer maintenance: compatibility backfills run in interruptible
background batches, so older indexes no longer delay server startup. - Operations:
/healthis available outside any configured base path.
Generated pages avoid stale caching while immutable assets use long lived
cache headers. - OAuth: provider discovery, token exchange, and profile requests now have
bounded request timeouts. - Video extraction: the
ytdlpextractor accepts anextra_domainslist
for additional supported sites. Archive.org pages no longer invoke it by
default. - Configuration documentation: settings show their server or client scope,
environment overrides appear earlier, and config path precedence is clearer. - Nix: the flake uses the upstream NixOS Hister module, and dependency
hashes are maintained throughnix-update. - Dependencies: Go modules, npm packages, container bases, Nix inputs, and
GitHub Actions were updated.
Bug Fixes
- Local files pass through the normal processing chain during reindexing, and
Markdown snapshots remain classified as local files. - File statistics include global documents in multiple user mode.
- Preview content and archived versions resolve by exact document identifier
instead of selecting another user's copy of the same URL. - Partial rule updates preserve any omitted rule groups.
- Search suggestions no longer contain duplicates.
- Input method composition no longer starts searches before text composition
completes. - Leaving the search page no longer displays an expected WebSocket error.
- PostgreSQL semantic search rejects unsupported vector dimensions before
attempting to create the vector store. GET /mcpreturns method not allowed instead of the web application.- The container health check works when Hister uses a nonroot base URL.
- Safari browser extension origins are accepted by CSRF validation.
- Unrelated files whose names resemble language indexes are ignored.
- Bleve keeps its established open file allowance, avoiding index failures
without reducing performance.
Backward Compatibility Notes
The following changes require attention when upgrading from v0.18.0:
- Nix flake:
x86_64-darwinis no longer exported because Nix 26.11 does
not support it. The Hister flake module now relies on the upstream
services.histermodule supplied by nixpkgs.
v0.18.0
New Features
Search Suggestions and Sorting
The web search box now suggests matching queries from search history, recent searches, aliases, fields, sort options, and facet values. Facet suggestions show result counts. The panel supports keyboard and pointer use, can be resized, and remembers its height.
A shared server schema keeps fields, facets, values, and sort options consistent across search clients. Queries can use sort:relevance, sort:date, sort:visits, or sort:domain. Prefix the value with a minus sign to reverse the order, as in sort:-date.
Social and Discussion Extractors
New Twitter, Bluesky, Reddit, and Discourse extractors preserve useful post, comment, reply, author, date, and reaction data. Twitter and Bluesky can split feeds and threads into separate post documents. Twitter also expands known shortened links to their original destinations.
Documents can now include JSON LD metadata. A new extractor SDK and registry define explicit results, capabilities, and optional context support.
Terminal Result Details
The terminal interface now has responsive workspaces and a readable result details pane. Wide terminals show the list and preview together. Narrow terminals use the full workspace. Text layout accounts for terminal cell width.
Local File Format Imports
hister import file can now import PDF, DOCX, Markdown, Org mode, valid UTF 8 text, and other supported local formats. It accepts individual files or directories. Extraction runs in the command line process and creates remote snapshots, so the server does not need access to the source files.
Service and Browser Imports
Hister can now import incrementally from Linkding, Readeck, and wallabag. Stored content is used when available, with a page download fallback. Browser imports can start at --start-date.
Bulk Document Updates
The new hister update command changes the owner, label, title, or language of documents matched by a search query. It supports dry runs, asks for confirmation, reports conflicts, and keeps file ownership rules intact.
Complete History Timeline
The history page now shows counts for all indexed or opened history. Recent days and older months are grouped for easier browsing, with daily details for a selected period.
Persistent Web Sessions
Web sessions are now stored in the configured SQL database. They survive server restarts, use rolling expiration, and are revoked at logout. Cookie security follows the configured server URL.
Structured MCP Results
The MCP endpoint now targets protocol version 2025-06-18 and defines output schemas for every tool. Results separate trusted metadata from untrusted source content, remove invisible control characters, and include safe handling guidance. Search callers can select full document fields.
Enhancements
- Search interface: search duration is visible, index statistics are clickable, empty indexes show quick start help, errors are clearer, and recent searches are easier to identify and clear.
- Query history: results promoted by query history are marked and can be forgotten for that query. Existing rules remain intact.
- Command line: help now groups commands by scope.
check-updatereports new releases, andcrawl urlscan filter or count stored crawl URLs. - URL input:
hister index --inputaccepts a file or standard input and creates a persistent job. - Files: watched directories can apply labels. Cleanup removes local documents that no longer match configured directories. File previews verify ownership and directory rules.
- Batch imports: clients split batches at the configured request limit and report documents that are too large.
- Browser extension: the extension supports access tokens and copied browser sessions. It refreshes changed cookies after authentication failures and submits pending page changes when a tab closes.
- Web interface and docs: the header is more compact, the initial color scheme is configurable, accessibility is improved, and documentation now has search and clearer navigation.
- Operations: profiling has an explicit setting. Container builds are smaller, and release images use Buildx Bake for supported architectures.
- Crawler rendering: capture delay is shared by rendered backends and is now supported by Chromedp.
- Terminal compatibility: command output and the TUI now use Charm v2, respect terminal color support, and preserve custom hotkeys.
- Dependencies: Go modules, npm packages, Nix inputs, container bases, and GitHub Actions were updated.
Bug Fixes
- Failed embedding jobs no longer starve later queued documents.
- Index metadata access is safe during concurrent indexing.
- Batch requests respect the server body limit and report oversized documents.
- Field specific alternatives work correctly, and URL wildcard matching ignores letter case.
- Browser imports count URLs correctly and handle empty history databases.
- OAuth provider scopes are combined with default scopes.
- Copied extension sessions use consistent expiry, and the background script works in browsers that require a classic script.
- Sensitive content matches are no longer exposed in debug logs.
- File indexing continues after an invalid symbolic link.
- Opened history responses are decoded correctly.
- Mastodon pages without a detected post are no longer stored.
- Optional versioning rules remain optional, and prioritizing a result keeps the full rule set intact.
- The middle pointer button still opens results while suggestions are visible.
- Database conflict clauses now qualify column names correctly.
- The configured color theme is applied before the page becomes visible.
Backward Compatibility Notes
The following changes require attention when upgrading from v0.17.0:
- Login sessions: web users and extensions that copied a browser session must sign in once after upgrading. Access tokens are unchanged.
- File preview API:
/api/filenow accepts a documentidinstead of an absolutepath. Search and history results include this identifier. - MCP output: tool results now use schema version
1.0. Untrusted source values are understructuredContent.untrusted_content. Text output contains the same JSON and a security notice. - Extractor implementations: custom extractors must use the new SDK result constructors, capability declarations, and registry.
- URL list input:
hister index --url-listremains as a hidden deprecated alias. Usehister index --input; it also accepts-for standard input.
v0.17.0
New Features
Unified Imports
The import interface is now grouped under hister import. In addition to local
files and browser history, Hister can import bookmarks and archived content from
Linkwarden, Karakeep, and Shaarli. Service imports preserve source dates and
metadata, apply source specific or overridden labels, fetch missing page content
and favicons, insert documents in batches, and continue from the latest imported
update after an interruption.
Persistent Crawl Job Tooling
Persistent indexing jobs can now start from URL lists, and browser history
imports use the same resumable crawl workflow. Jobs can be resumed with only a
job ID. New crawl show, crawl errors, and crawl queue commands expose job
state, failed URLs, queue contents, and queue counts.
Expanded Query Language and Search Filters
Documents now track an updated timestamp alongside their original added time.
Search queries support added and updated filters with relative or absolute
dates, visit count ranges, standalone wildcard expressions, and exact phrases
within specific fields. The search interface represents filters directly in the
query, making filtered searches easier to understand, edit, and share. Existing
index entries have their updated value backfilled from their added time on
startup.
Durable Semantic Indexing
Embedding work now uses a persistent queue and deduplicates pending documents.
Token measurement is more precise, requests reserve capacity below provider
limits, and batch size plus request timeout are configurable. Chunk construction
also uses document structure and more useful metadata.
Extensible File Types and DOCX Support
Local file handling now uses an extensible file type interface. DOCX documents
can be indexed with extracted text and metadata, and the documentation explains
the supported local formats and configuration.
Chinese, Japanese, and Korean Search
Language detection and language specific indexes now support Chinese, Japanese,
and Korean documents.
GitHub Issue and Pull Request Extraction
The GitHub extractor now understands issue and pull request pages. It indexes
the title, body, open date, comments, repository metadata, and useful page text.
Crawler Proxy Support
HTTP and SOCKS5 proxy URLs can now be configured for every crawler backend. The
new --proxy option also allows crawler backed index and import operations to
override the configured proxy, including robots.txt requests.
RFC Dataset
A new RFC dataset and fetch command make it easier to build a searchable local
collection of Internet standards.
Qutebrowser DevTools Companion
The new hister companion qutebrowser command connects to the Qt WebEngine
DevTools interface and indexes rendered tabs from qutebrowser. It accesses the
final document after client rendering without requiring a page userscript.
Enhancements
- Search interface: sort choice is preserved in the URL, result index
counts are visible, and the search input remains available while browsing
results. - Semantic results: returned chunks are limited per document, metadata is
richer, and index metadata is stored with the index itself. - Language aware indexing:
indexer.keep_stopwordscan retain common words
while preserving language detection, normalization, and stemming. - Crawler efficiency: already skipped URLs no longer incur the configured
crawl delay. - Import performance: imports use bulk insertion, retain useful imported
titles and processed state, apply labels, and retrieve missing favicons. - Favicon delivery: favicons are served through a dedicated endpoint rather
than being embedded in document responses. - Mastodon extraction: remote toot URLs are resolved to their canonical
locations before indexing. - Command line workflow: search supports
--sort, delete supports--yes,
list files supports--relative, and command descriptions are clearer. - History: filtering now happens on the server and the selected filter is
represented in the page URL. - Rules interface: multiple rules can be entered at once and rule lists use
descending order by default. - Browser extension: pages may be submitted as public documents, Firefox
mobile capability differences are handled, and automatic indexing accepts
only HTML, XHTML, or plain text documents. - Website and documentation: the landing page, onboarding, navigation,
accessibility, configuration reference, SEO metadata, and 404 page have been
refreshed. - Operations: debug profile endpoints are available at debug log level, a
sample systemd unit is included, and crawler failure codes are retained for
inspection. - External tools:
yt-dlpinstallation uses upstream releases and
concurrentyt-dlpextraction is limited. - Dependencies: Go modules, npm packages, Nix inputs, GitHub Actions, and
browser extension dependencies were updated.
Bug Fixes
- SQLite semantic search uses cosine scoring consistently.
- MCP search returns semantic results correctly.
- Embedding requests avoid provider token limits and reduce batch size when a
provider reports an oversized request. - Index resources are closed when initialization fails, absent documents are
not deleted again, and document insertion avoids repeated lookups. - Startup only warns about reindexing for analyzer changes when analyzer
configuration is present. - Stored previews reject meta refresh redirects.
- Document domains no longer include ports, and invalid favicon schemes are
ignored. - Public mode history is available to authenticated users and response headers
are no longer written more than once. - Field specific phrase queries and a single wildcard query produce the
expected matches. - Imports preserve existing titles when extraction finds no replacement, and
the JSON input scanner accepts larger records. - The browser extension avoids unavailable command APIs on Firefox mobile and
uses a compatible badge text color. - Search result and history titles and URLs retain usable space in narrow
layouts.
Backward Compatibility Notes
The following changes require attention when upgrading from v0.16.0:
- Import commands:
hister import-browserhas been replaced by
hister import browser. File and directory imports now require the source
subcommand, as inhister import file INPUT. Imported JSON exports retain
their processed content without running extractors again. Browser imports now
use persistent crawl jobs and can prompt to resume an unfinished job. - Date behavior: the structured
date_fromanddate_tosearch parameters,
facet date ranges, export date flags, date sorting, history ordering, and
history RSS dates now use the documentupdatedtimestamp instead of
added. The web interface no longer readsdate_fromordate_tofrom its
page URL. Putadded:orupdated:expressions in theqparameter instead.
For newly indexed local files,addedis the indexing time andupdatedis
the file modification time. - Favicon responses: search and history results return
favicon_keyinstead
of embedding stored favicon data. API clients should fetch the image from
/api/favicon?key=FAVICON_KEY. Legacy inline favicon values remain readable. - Domain normalization: newly indexed and reindexed documents store only the
hostname indomain, without the port. Queries and integrations that expect a
host and port value must be adjusted. - Semantic search: SQLite vector stores migrate from Euclidean distance to
cosine distance. Customsimilarity_thresholdvalues may need adjustment.
When omitted from configuration, the default context length changes from
4096 to 512, chunk overlap from 128 to 64, and embedding concurrency from 10
to 2. - Browser extension submissions: the extension no longer submits images,
videos, or other unsupported document types. This content type restriction
also applies to manual submission.
v0.16.0
New Features
Public Search Mode
Hister can now run in public mode, allowing unauthenticated read-only access to
global search results and previews while keeping write operations protected.
This makes it possible to publish a shared or community search instance without
exposing private user data. Public mode is documented across configuration,
server setup, user handling, and the new public search blog post.
Documentation Datasets
Hister now ships dataset metadata and tooling for importing common reference
documentation. New datasets cover Go standard library, PowerShell, Python, Rust,
Node.js, and refreshed MDN datasets. Compressed dataset imports are supported, so
pre-built documentation archives can be imported directly.
StackExchange Extractor
The StackOverflow extractor has been replaced by a generic StackExchange
extractor. It supports StackExchange-style question pages more broadly and
indexes answers with useful metadata.
Markdown and Org Mode Extractors
Local Markdown and Org files now render as HTML previews. Markdown titles are
extracted automatically, Markdown syntax is stripped from indexed text, and Org
files are rendered through go-org for cleaner previews.
MCP History and Capability Reporting
The MCP endpoint now exposes a get_history tool for recently indexed pages and
opened search result history. MCP search supports date filters, and MCP clients
can discover whether semantic search is enabled on the current Hister instance.
The MCP documentation and examples have been expanded accordingly.
Browser Extension Hotkeys and Indexing Indicators
The browser extension now includes configurable commands to manually index the
current page, disable indexing for the current page, and disable indexing for
the current domain. Badge feedback indicates success or failure. The extension
can also show whether the current page has already been indexed, with a popup
setting to control the indicator.
Import and Export Improvements
Imports now support HTML files, multiple input files in a single command,
compressed JSON archives, directory imports for supported local file types, and
optional user IDs. Import and export commands can be limited by date range.
Browser history import gained a redesigned interactive flow and a --backend
flag for selecting the scraping backend.
Search JSON Output
The /search endpoint now supports format=json, making it easier to integrate
Hister search results with scripts and external tools.
Search Sorting
Search results can now be sorted by newest date, domain, and most visited pages.
Documents track an add_count field that records how many times a URL has been
indexed, and the web UI exposes this as the "Most visited" sort mode.
History RSS Feed
The history endpoint and history UI now expose an RSS feed for recently indexed
documents.
Enhancements
- File indexing startup: initial file indexing now runs in the background so
server startup is not blocked by watched directory indexing (closes #499) - Directory indexing queue: watched directory indexing uses a proper queue,
logs queue errors, and avoids duplicating file indexing logic - Sensitive content override: CLI indexing can explicitly allow sensitive
content when requested - Semantic search: embedding generation has a concurrency limit, semantic
result counts are more accurate, and shared plus user-specific vector results
are merged - History view: the history timeline has been redesigned, supports loading
more items, shows when a day has unloaded items, and displays indexed version
counts - Search UI: the results layout, home layout, dark theme, add entry form,
search input, and shared chrome have been refined - Facets: type facets were added, inactive facet categories are hidden when
they only have one candidate, and facet rendering was refactored - Preview and history integration: history supports diff previews for
versioned documents, preview links are proper anchors, and previously opened
results stay visible after unpinning - Configurable branding: the web UI title and subtitle can now be configured
- Logging: log output format and log file path are configurable, and panic
stack traces are available in debug logs - Environment variables: environment variable overrides now preserve typed
values instead of treating everything as a string - Terminal UI: the TUI package now lives under
cmd/tui, includes updated
command documentation, and supports result sorting - CLI structure: the command line implementation was split into a dedicated
cmdpackage for easier maintenance - Docker image:
yt-dlpis now installed in the Docker image - Dependencies: Go, npm, GitHub Actions, Nix, GORM, SQLite, and PDF related
dependencies were updated
Bug Fixes
- Lobsters extraction works with the updated Lobsters HTML structure
- Document labels are added and preserved correctly
- Opened history filtering and history list layout no longer squeeze content
- Result text wraps correctly in the web UI
- Description lines no longer force table scrolling
- Base URL handling is more consistent for app URLs, CSRF checks, and invalid
base URL values - The preview panel is cleared when a search returns no results
- Reindexing no longer stops when an extractor aborts one document
- Lazy loading works when the previous page returned fewer than the configured
page size - Database migrations no longer unintentionally update
UpdatedAt - PDF parsing panics are recovered, and PDF dependency updates address memory
issues - Versioned document diffs use the newly stored document when calculating
changes - Batch operations handle user IDs correctly
- Extra documents no longer increment the parent document add count
- Subtitle rendering is centered correctly
- Python dataset generation fetches only standard library documentation
- Nix paths and hashes were updated after Go and npm dependency changes
v0.15.0
New Features
Document Versioning
A new versioning rule type instructs Hister to track changes to a document each
time it is re-indexed. A diff-style changelog appears inside the preview panel so
you can see what changed between versions, and the preview endpoint returns the
full version history. Combine with a priority or skip rule to version only the
pages that matter to you.
Priority Rules
Priority rules have been reintroduced (closes #222). Documents whose URLs match a
priority rule are pinned to the top of search results regardless of their relevance
score. Rules can be created, edited, and sorted directly in the web UI.
PDF Indexing
Local PDF files can now be indexed with full text extraction. The browser extension
gains a new endpoint for sending PDFs directly to the server (closes #55). Indexed
PDFs are stored as a distinct document type and can be filtered with
metadata.type:pdf in search queries.
Faceted Filtering
Search results can now be filtered through facets that group documents by common
fields such as language, and domain. Facet counts update live as you refine your
query. Date-range filters have been moved into the same filters dropdown for a
unified experience, and a "load more" control expands facets with many values.
Embedded Video Extractor
A new extractor detects embedded videos (YouTube, Vimeo, and similar platforms)
on indexed pages and stores the embedding metadata as a dedicated document type.
Embedded videos are rendered directly inside the result preview panel, so you can
watch them without leaving Hister (closes #446).
Notion Extractor
A dedicated extractor for Notion pages extracts article content cleanly from
public Notion URLs, removing navigation chrome and other non-content elements.
Compressed HTML and Favicon Storage
HTML content and favicons are now stored gzip-compressed in separate files on
disk rather than inline inside the Bleve index. This significantly reduces index
size on disk and lowers memory pressure during search (closes #384).
Disable HTML Storage
A new config option lets you turn off full HTML storage and preview generation
entirely. Disabling storage trades preview functionality for a smaller footprint,
useful for bulk or headless indexing workflows (closes #440).
User-Specific Directory Indexing
The indexer.directories config now accepts a user field per directory entry.
Files under that path are indexed only for the specified user, making it easy to
share a single Hister instance while keeping personal file indexes private.
Label Editing from Results
Document labels can be edited inline directly from search result cards without
navigating away. The browser extension can also apply a one-off label to a
document at index time (closes #407). Updating a document no longer clears
previously assigned labels.
File Deletion Tracking
When a locally-watched file is deleted from the filesystem, Hister now
automatically removes it from the index (closes #230). Batch and single-document
deletions also clean up the associated HTML and favicon files from disk.
MCP Document Preview Endpoint
A new MCP endpoint exposes document previews to LLM agents and MCP-compatible
tools, complementing the existing MCP search endpoint.
Browser Import: Ladybird Support
The import-browser command now supports importing history from the
Ladybird browser.
Browser Import: Auto-Detect Database
The import-browser command now automatically detects the browser database file
path, so passing an explicit path is no longer required for supported browsers.
Enhancements
- Copy URL button: a copy-to-clipboard icon appears next to each result URL
- Preview extractor selector: switch between available extractors when viewing
a document preview without re-indexing - Rules table: columns are sortable; values are filterable; filter toggle
buttons are visually distinct from column headers; regexp validation runs before
saving a rule - CLI index flags:
--delay,--timeout, and--user-agentare now
available directly onhister indexwithout needing a config file - Configurable client timeout: the HTTP client timeout used during indexing
is configurable (fixes #429) - Standardized config file lookup: config file discovery follows a consistent
search order across all platforms (closes #424) - History autoscroll: the history view scrolls to keep the selected entry
visible (#427) - Admin profile version: the admin profile page now shows the running Hister
version (closes #409) - Log level aliases: common short aliases are accepted for log level values
(fixes #411) - Random tips: a rotating set of usage tips is shown on the front page when
navigating back from search results - yt-dlp diagnostics: clearer error messages and debug logging help diagnose
yt-dlp configuration problems - Batch crawl insertion: URLs discovered during a recursive crawl are
inserted in batches for better throughput - Extension popup header: the popup header is now a link to the configured
Hister server - Per-sub-index paging: each language sub-index is iterated independently to
prevent paging gaps in multi-language setups - Improved embedding handling: fallow and embedding request processing
is more robust
Bug Fixes
- Docker
BASE_URLenvironment variable can now be correctly overridden by a
config file (closes #442) - Search results now contain all queried terms rather than any of them
- HTML is no longer re-written to the indexer during a reindex run
- yt-dlp subtitle download no longer skipped when the sub language differs from
the original language (#429) - History page stops loading more entries when the last page has been reached
- "Show all" in history now scrolls back to the top
- Rule type filter no longer incorrectly excludes rules
- Debug-level init messages are correctly suppressed at higher log levels
- Result paging uses a valid sort key, fixing out-of-order pages
- Times are displayed in the browser's configured timezone
WARNING
If you are using Hister with a custom config file under Docker, make sure you have the listen address set to 0.0.0.0:4433 in your config file.
v0.14.0
New Features
Mastodon Extractor
A dedicated extractor for Mastodon detects Mastodon pages and indexes each
toot as its own separate document rather than one big blob of text. Works
with any Mastodon instance without configuration. Every toot gets a
metadata.type:toot field so you can filter toots in search queries.
Combined with a search alias (!toot → metadata.type:toot) this makes
finding past toots fast and convenient.
Metadata Query Filtering
Documents can now be filtered by arbitrary metadata fields using the
metadata.key:value query syntax. Extractors (including the newMastodon
extractor) populate these fields at index time.
Full Screen Preview
The split-screen preview can now be toggled into a full-screen mode that
occupies the entire content area (closes #401). The URL changes to
/preview/[id] so the view survives a page reload. Pressing the "view
result content" hotkey switches between split and full-screen. Full-screen
preview is also available on the history page.
Preview Panel on History Page
The history page now shows the same interactive preview panel as the search
page (closes #395). Hotkey navigation between history entries works the same
way as on the search page.
Infinite Scroll
Search results now load more pages automatically as you scroll to the bottom
of the list, removing the need to manually page through results (closes #1).
Image Lightbox
Images displayed inside the preview panel can be clicked to open a full-size
lightbox view.
Delete All Results
A new "delete all" action removes every document that matches the current
search query at once, without having to delete results one by one.
Keyboard Shortcut for Result Deletion
A dedicated hotkey deletes the currently focused search result directly from
the keyboard.
Quick Skip Rule
A skip-rule component lets you add a URL to your skip list directly from a
search result, available in both the web app and the browser extension
(closes #380).
robots.txt Support
The crawler and hister index now respect robots.txt by default (closes
#386). A new ignore_robots_txt config option disables this check when
needed.
Document Labeling
Documents can be tagged with custom labels when indexing from the CLI
(--label) or from the browser extension. Labels are stored as metadata
and can be used in search queries (closes #156, #373).
Editable Rules and Aliases
Existing skip/priority rules and search aliases can now be edited in the
web UI instead of having to delete and recreate them (closes #270).
Exact Phrase Matching
Multi-word queries now also attempt an exact phrase match across title and
text, so searching for open source surfaces pages that contain that exact
phrase more prominently (closes #394).
Extractor Templates and Extra Document Creation
Extractors can now supply a custom preview template, giving each content
type its own presentation in the preview panel. Extractors can also produce
additional sub-documents from a single page (used by the Mastodon extractor
to create one document per toot). A template scaffold is included to make
writing new extractors easier.
WebDriver BiDi Crawler Backend
A new bidi crawler backend uses the W3C WebDriver BiDi
protocol to drive an already-running browser over a WebSocket connection.
Unlike the chromedp backend, it does not launch a browser process: you start
the browser yourself (headless or not) and point Hister at it:
# Firefox
firefox --remote-debugging-port 9222
# Chrome / Chromium
chromium --remote-debugging-port=9222crawler:
backend: bidi
backend_options:
host: '127.0.0.1'
port: '9222'
capture_delay: 1.5 # extra seconds to wait after load for JS renderingSupported by Firefox (≥ 102), Chrome/Chromium (≥ 106), and Edge. Options:
socket (full WebSocket URL), host, port, capture_delay. The crawler
reuses a single BiDi session for all URLs in one hister index run, making
multi-URL indexing significantly more efficient than opening a new browser
session per URL (closes #284).
Enhancements
- Resizable preview panel: drag the divider in split-screen view to
adjust the panel width; the chosen width persists across sessions - History hotkey navigation: keyboard navigation between history entries
on the history page - Secondary date sorting: when search scores are equal, results are
sorted by indexed date - yt-dlp multi-language subtitles: configure which subtitle languages
the yt-dlp extractor indexes - MCP date filtering: MCP search requests can now be filtered by date
- Semantic search chunking: punctuation-based boundaries used for
chunk splitting, improving relevance for sentence-level queries - OIDC enhancements: userinfo endpoint is now configurable for
providers without auto-discovery (#279); password login can be disabled
when OAuth is the only configured auth method - Clearer CLI error messages: client-side HTTP errors now explain the
problem in plain words and suggest which flag to use (#400) - Duplicate rule prevention: the server and UI both reject duplicate
skip/priority rules and aliases (#399) - Deletion error feedback: errors during document deletion are now
surfaced in the UI - Rules/aliases UX: input fields moved above their respective lists
- Version string in server log: the server start message now includes
the version number (#372) - Silent WebSocket disconnect: closing the browser tab no longer shows
a connection-error message
Bug Fixes
- Search terms are now properly escaped before query execution
- Focused result index computed correctly when priority results are present
- Phrase queries are only applied when no field-specific terms exist in the
query /api/deleterequests from the browser extension are now acceptedtextandhtmlfields always included in search results (#374)- Language field included in document search results
- OIDC scopes correctly forwarded to the provider (#371)
- Auth page is scrollable on small screens (#370)
- TUI client now passes the access token to the server (#368)
- Optional peer dependencies no longer excluded on non-Linux platforms (#299)
v0.13.0
New Features
Semantic Search
Full vector search support via sentence-embedding models. Documents are chunked
and embedded at index time; search queries are embedded at query time and ranked
by cosine similarity. Two storage backends are supported:
- SQLite (default, via bundled
sqlite-vec) zero extra infrastructure required - PostgreSQL with
pgvectorauto-selected when the database is Postgres
Configure the embedding API endpoint, model, dimensions, and chunking parameters
in the new semantic config section. Semantic search is opt-in and off by default.
Relevance scores are shown alongside results when semantic search is active.
OAuth / SSO Authentication
OAuth 2.0 and OpenID Connect (OIDC) providers can now be configured as login
methods. Add one or more entries to the new server.oauth config section with
client_id, client_secret, configuration_url (for OIDC auto-discovery), or
manual auth_url / token_url, and optional scopes. Multiple providers can be
active at the same time alongside the built-in username/password login.
MCP Server
Hister now exposes a Model Context Protocol
endpoint at /api/mcp, enabling LLM agents and MCP-compatible tools to search
the index directly.
Persistent Crawler State Management
Recursive crawl jobs (hister index -r) are now stored in the database and
survive interruptions. Each job gets a unique ID (auto-generated or set via
--job-id). Pass --job-id <id> without --recursive to resume an
interrupted crawl from exactly where it left off, including original validator
rules and visited-URL counts.
New Extractors
- Wikipedia extracts the article body and infobox, rewrites relative links, and sanitizes the output
- GitHub project extracts repository descriptions and README content from GitHub project pages
- Lobste.rs dedicated extractor for Lobste.rs story and comment pages
- yt-dlp extracts video metadata (title, description, channel) from video pages via yt-dlp
- JSON-LD surfaces structured metadata (
@type,headline, description) from pages that embed JSON-LD
All extractors now expose a Description() method, and an extractor information
page is available at /extractors in the web UI.
OpenSearch Suggestions
The server now serves an OpenSearch suggestions endpoint (/api/suggest),
allowing browsers to display search-as-you-type completions when Hister is
configured as a search engine.
Enhancements
Crawler Backend for All Index Operations
The --backend flag (and --backend-option) is now available on both
hister index (plain and --recursive) and hister import-browser, allowing
a headless Chrome/Chromium backend for JavaScript-heavy pages without running a
full recursive crawl:
hister index --backend chromedp https://example.com
hister import-browser --backend chromedp --backend-option exec_path=/usr/bin/chromiumHeaders and cookies can also be injected per-invocation:
hister index --header "Accept-Language=en" --cookie "session=abc; Domain=example.com" https://example.comCookies use standard Set-Cookie format with a required Domain attribute.
CLI Search Improvements
--limit Nflag caps the number of results returned--fieldsflag selects which document fields to include in output--htmlflag includes raw HTML content in the output- Paging support added to both CLI search and
list-urls list-urlsnow fetches results from the server by default;--offlineconnects directly to the index without a running server
Quoted Field Queries
Field-qualified queries now support quoted values, enabling correct deletion and
lookup of URLs that contain spaces (common on Windows file paths):
url:"file:///C:/Users/My Documents/notes.txt"
Preview Panel Polish
- Preview title is now clickable (opens the result URL)
- Preview panel maximises available content width
- JSON-LD metadata surfaced inside the preview panel
- Dark theme font colors fixed in preview popup
NixOS / Nix Module
systemdandlaunchdhardening applied to the Hister service units- New
services.hister.environmentFileoption for secrets injection openFirewallnow requires explicit opt-inservices.hister.configrenamed toservices.hister.settings
Other
- Executable size reduced ~70 MB by switching to a trimmed
lingua-gofork - Sensitive content rejection errors surfaced in the browser extension
--verboseflag onhister deletelists matched URLs before deleting- Priority result deduplication now copies body text from the original result
/suggestendpoint protected by auth middleware andSec-Fetch-Siteheader check- Version information included in the MCP endpoint response
- Timezone data bundled into the binary for environments without a system
tzdata
Bug Fixes
- File URLs (
file://) now handled correctly in the UI for both opening and deletion (#362) - Browser extension authentication documentation corrected (#366)
- URLs no longer lowercased during query building, preventing mismatches on case-sensitive paths
- History view correctly filtered per-user in multi-user mode (#314)
- Token authentication middleware now respects
NoAuthflag (#348) - Documents with no HTML content no longer attempt HTML extraction (#351)
- Extension no longer resubmits documents after a
406 Not Acceptableresponse - Priority results correctly deduplicated against standard results
- File indexing fixed on Windows
- Wide tables no longer overflow the preview panel
- Score field populated correctly in search responses
aws_access_keysensitive content pattern tightened to reduce false positives- Home-manager service units correctly gated on host platform in Nix module