Sources and Mirrors

Sources and mirrors

Wiki home · related: The cycle · Troubleshooting · full text: source modules

Aurora never reads Confluence or Jira while it builds cards. It first makes a mirror — a deterministic copy in Sources/ — and builds from that. Your own evidence lives in Raw/.

flowchart LR
  C["Confluence"] -->|"sync:confluence"| MC["Sources/Confluence"]
  J["Jira"] -->|"sync:jira"| MJ["Sources/JIRA"]
  W["Websites"] -->|"sync:web"| MW["Sources/Web"]
  R["Raw/<br>contract · laws · meetings ·<br>customer · project · corrections"] --> B
  MC & MJ & MW --> B["agent:build<br>parsing into cards"]
  R -->|"docx · pdf · xlsx · pptx<br>kb:ingest-office"| R

Loading

Mirrors: deterministic, incremental, never edited by hand

• The same source gives the same file, byte for byte. Otherwise git diff stops being evidence and sync:audit cannot check the state. A repeated sync makes no noise in git.
• Incremental. A Confluence page with the same version, converter and path, and the file in place, is not downloaded again; once a week and with --force the mirror is passed in full.
• Confluence attachments as files. PDFs and documents from view-file, images and attachment links are downloaded into <page>_assets/ and the text links to them; executables, scripts and html (exe, bat, sh, js, py, html and alike) and dot-named files are never downloaded, and what was skipped stays a "not downloaded" row with a link to the original in the ## Вложения страницы section. What to download is set in atlassian.confluence: attachments: all | images | none (default all) and attachment_max_mb (default 100, 0 = no limit); both are edited in the panel, "Project settings" → the project form, below the Confluence address and space. What is downloaded goes into the mirror's git, so for sources with large documents (laws, regulations) set a lower limit or "do not download". A file skipped by mode or size is also a "not downloaded (reason)" row. The sync summary has a line Вложения: скачано N · пропущено по типу M · по размеру K · ошибок E (downloaded / skipped by type / by size / errors); changing these settings re-reads the already synced pages that have attachments.
• Written only by sync. Do not edit Sources/ — the next sync would overwrite you; put your own words into Raw/corrections/ (kb:correct) instead.
• A failed sync keeps what it did not reach. If the server timed out halfway, the pages the walk did not visit stay in the state, and --prune refuses to delete when the export had errors or the "extra" share is large.

The bundled modules

Module	Kind	Role	Mirror	Command
confluence-dc	wiki: folders repeat the page tree, a page with children → folder + index.md	documents	Sources/Confluence/	sync:confluence
jira-dc	board: a file per issue, the key is the name	tasks	Sources/JIRA/	sync:jira
web	board: a file per saved page	documents (trust per link)	Sources/Web/	sync:web

More products (Notion, SharePoint, Confluence Cloud, YouTrack, GitLab Issues) are added as a folder in connectors/ with a connector.json; see source modules.

Settings and secrets

What	Where
base URLs, spaces, sync_roots (page ids to walk from), project key, JQL	aurora.config.yaml (committed)
CONFLUENCE_PERSONAL_TOKEN, JIRA_PERSONAL_TOKEN (or user + password)	.env.aurora.local (git-ignored)

Sync scripts talk to Confluence and Jira directly over REST. The Atlassian MCP in your editor does not help them: it does not hand out its credentials. Check a token: python3 .aurora/scripts/jira_export.py --limit 1.

A token goes only to its own server. The address in aurora.config.yaml is shared by the team, and swapping it in a pull request would carry the token away. So the REST client sends the key only to the server in base_url: an absolute address that comes from the server (_links.next, an attachment) with another host is refused, and a redirect to another host goes without Authorization. The key is never sent over plain http:// (your own computer excepted). The engine also remembers the server after the first successful reply (.aurora/state/trusted-hosts.json, not in git): a different base_url afterwards is refused, with one line saying so, until you accept it. Accepting is one line in .env.aurora.local: AURORA_CONFLUENCE_SERVER=https://wiki.example.com or AURORA_JIRA_SERVER=https://jira.example.com (the same lines pin the server in advance — and for a project that has never synced, nothing is remembered yet, so set them there). AURORA_ALLOW_HTTP=1 allows plain http:// for a server on a closed network. Model gateway keys and the Git password are protected differently: a redirect never carries them to another host, a saved key is not carried over to a changed address, and the credential helper answers only the repository's server; plain http:// stays allowed there (a local Gitea or gateway on a closed network). The website crawl token (WEB_PERSONAL_TOKEN) goes only to the host of the start page it is used with; a page added to the shared list would get it, so the engine remembers the origins (scheme, host and port) that answered in the first run (.aurora/state/trusted-hosts.json, not in git; your own computer is never remembered and is never cut off by the binding; a dry run remembers too): a page added later on another host or port does not receive it, and the log says how to allow the new address. The run ends with one line: "токен привязан к …" (bound; lists the hosts that did not answer, they will not get the token) or "токен не привязан …" (not bound: nobody answered or the binding could not be written; the token still goes to every host of the list); kit:doctor warns "token present, no binding". To bind the hosts before the first run (or to accept a new one), list them in AURORA_WEB_TOKEN_HOSTS=wiki.example.com,…: the line replaces the remembered list (name all hosts, the old ones too) and no other host receives the token (your own computer is the exception: it always gets it). A bare host matches any port of that host; wiki.example.com:9443 and https://host[:port] match only their own (the one exception, as for the start page: http://host also admits https://host).

The website crawl stays within bounds: only http:// and https:// addresses are fetched (file://, ftp://, data: are refused), and on every hop, redirects included (at most 5), the host name is resolved and the crawl refuses anything that is not a public address: your own computer (127.x, ::1), private networks (10.x, 172.16–31.x, 192.168.x, 100.64.x), link-local (169.254.x, the cloud metadata address) and ::ffff: forms of those. A refusal is one line in the report (запрещено границами коннектора: …) and is not retried. An internal wiki is allowed explicitly in the web: block of aurora.config.yaml: allow_private: [wiki.example.com, localhost:8080] (a bare host matches any port, wiki.example.com:8080 only that port, subdomains are not included). The list only lifts the address check; it does not add the host to the token binding above, and the scheme is checked always. The address is checked again at connect time, and the socket goes to the address that was checked (so a name that answers with a public address first and 127.0.0.1 second, DNS rebinding, is refused). A proxy from the environment or the system settings (HTTP_PROXY/HTTPS_PROXY, NO_PROXY) is honoured: for a request that goes through the proxy the name is not resolved locally (the proxy resolves it, so the address check cannot see it; the scheme and literal addresses, 10.0.0.5 or 2130706433, are still refused), and a host in NO_PROXY takes the direct path with the full check (localhost and *.localhost are refused even through a proxy; a trailing dot, localhost. or 127.0.0.1., changes nothing: the name is brought to the form the transport sends (IDNA, lower case, no trailing dots) in every check, so a soft hyphen or ¹²³ inside it hides neither an address nor an internal name, and allow_private entries are read the same way, as is kit:doctor; a name that cannot be encoded is refused as unparsable. Through a proxy the check is stricter: a name with characters outside a-z0-9._- (an IPv6 literal aside), 127.0.0.%31 included, is refused, because the proxy may read it differently than we do (write such a name as punycode, xn--…), and the clearly internal names are refused too: localhost.localdomain, ip6-localhost, ip6-loopback, metadata.google.internal, the suffixes .local, .internal, .localdomain, and a name with no dot at all; allow_private lifts them. Known limit: a name such as 127.0.0.1.nip.io cannot be recognised without resolving it, so through a proxy it passes the check, while on the direct path its address is seen and refused). Response size and time are limited too: a page up to 5 MB, an attachment up to 50 MB (constants PAGE_MAX_BYTES and ASSET_MAX_BYTES in scripts/web_export.py), read in chunks. A Content-Length above the limit is refused before the body is read, a stream without it is refused at the limit, and reading one response body has a total time of 2× the timeout, counted after the headers arrive (a server that sends a byte a second does not hold the run; the wait for the headers is limited only by the timeout of each read). A refusal is one line in the report (слишком большой ответ (> N МБ) or истёк общий срок), is not retried, and an attachment is never left on disk as a partial file. The page encoding is detected, not assumed: a byte-order mark, then the charset of the Content-Type header, then <meta charset> or <meta http-equiv="Content-Type"> in the first 4 KB, then strict UTF-8, and for Cyrillic pages a guess among cp1251, koi8-r and cp866 by the share of common letters (a candidate whose Cyrillic letters sit right next to Latin ones is dropped: that is a Spanish or Polish single-byte page read as cp1251, not Russian); the header wins over <meta>, and an unknown or false declaration is skipped, not fatal. Only web-page encodings are accepted in a declaration (utf-8, cp125x, koi8, iso-8859-x, cp866, mac-*, the Chinese, Japanese and Korean families); punycode, utf-7 and the like are skipped, and a declared utf-16/utf-32 over ASCII-compatible bytes is read as utf-8, whatever the alias. If nothing fits, the unreadable bytes are replaced and the run prints кодировка не определена next to the page and adds the mark to the page's status in the mirror table (обновлена · кодировка не определена); such a page is not cached, so the next run says it again instead of answering 304 silently. An IPv6 host is written in brackets: allow_private: ["[::1]"]. If your DNS returns fake addresses (198.18.x.x, the fake-IP mode of TUN and enhanced proxy clients), every external site looks like a private network: list those hosts in allow_private or bypass the proxy for them.

The crawl keeps to one origin: a link or an attachment is taken only if its scheme, host and port equal the page's (a link from an https:// page to http:// of the same host on the standard port is upgraded to https://, never fetched in clear text; a link to another host or port is dropped; http://host → https://host is allowed too). Pages, crawl links and attachments all go through the same scheme and address check as the start page, on every hop. The token goes only to pages of its own source within the token host list and never to an attachment; a redirect to another source carries no secret headers. The body of a redirect (3xx) response is not read at all, so a server that sends a redirect and an endless body does not hold the run. kit:doctor names the web: sources that use allow_private and the pages with a private address literal and no permission (names are not resolved) as a warning, not an error.

Checking a mirror

Command	Finds
sync:audit	MISSING, ORPHAN, MOVED, COLLISION, stale state, foreign files, headers with a repeated field or a forged trusted
sync:diff	drift — a source changed after the knowledge was built on it
sync:jira-status	issues closed while the requirement is not; work without requirements; links by mention
kit:remap-sources, kb:repair --gone-sources	a page moved: cards' source: follows it by page_id or document code

Raw/ is evidence

Contracts, laws, meeting transcripts and customer material go into Raw/{laws,contract,customer,project,meetings, examples,corrections}. By agreement it is immutable and counts as trusted without proof. Office formats are turned into Markdown transcripts next to the original by kb:ingest-office; a scan is read by a vision model if one is configured (there is deliberately no default model for scans — a guessed name gives a coherent fabrication).

---

This wiki is generated from the wiki/ folder of the repository — edit the pages there, changes made here are overwritten. · RU-Home