Skip to content

Development Guide

Jump to bottom
zach115th edited this page Jul 31, 2026 · 9 revisions

Development Guide

Dev loop

After editing a Jinja template

docker restart iriswebapp_app

Gunicorn caches templates; auto_reload = False in the dev compose. A hard browser reload alone is useless — the server keeps sending the cached template.

After editing Python source (volume-mounted)

docker restart iriswebapp_app iriswebapp_worker iriswebapp_ai_worker

Python files under source/app/ are volume-mounted — edits are live in the container. A restart picks them up without a rebuild.

Exception: if you edited docker/webApp/iris-entrypoint.sh or gunicorn settings, you need a rebuild (the entrypoint is baked into the image).

After editing a static asset (ui/public/assets/)

Static assets are baked into the image at build time, not volume-mounted. Two steps:

# 1. Deploy to running container immediately
docker cp ui/public/assets/<path> iriswebapp_app://iriswebapp/static/<path>
# Always use PowerShell — Bash/MSYS translates paths incorrectly

# 2. Also edit the source file so the fix survives the next rebuild

Verify the cp landed before restarting:

docker exec iriswebapp_app grep -c '<something>' /iriswebapp/static/<path>

After editing a Vite-bundled JS file (ui/src/pages/)

cd ui && npm run build
docker compose -f docker-compose.dev.yml build app worker ai_worker
docker compose -f docker-compose.dev.yml up -d --force-recreate --no-deps app worker ai_worker

Bundled JS (e.g. case.asset.js, case.rfiles.js) is compiled by Vite → baked into the image. No shortcut — full rebuild required.

Exception: ui/public/assets/js/iris/case.time.js is a static file (uses Vite publicDir), so it deploys via docker cp like any other static asset.

After adding an Alembic migration

docker exec iriswebapp_app alembic -c /iriswebapp/app/alembic.ini upgrade head

Or just restart — db.create_all() runs before Alembic and handles new tables. Column additions on existing tables still need Alembic to run.

After editing the nginx conf

nginx conf is baked into the nginx image with envsubst substituting ${VAR} placeholders. Do not docker cp the raw source file — it contains literal ${VAR} which crashes nginx.

docker compose -f docker-compose.dev.yml build nginx
docker compose -f docker-compose.dev.yml up -d --force-recreate --no-deps nginx

Full image rebuild + DB-preserving recreate

docker compose -f docker-compose.dev.yml build app worker ai_worker
docker compose -f docker-compose.dev.yml up -d --force-recreate --no-deps app worker ai_worker

--no-deps keeps the DB and RabbitMQ containers running so no data is lost.

Upgrade (pull + rebuild)

git pull
docker compose -f docker-compose.dev.yml up -d --build --force-recreate

Always include --force-recreate — a plain up --build can leave worker / ai_worker on the old container while only app is rebuilt, causing NotImplementedError in task_hook_wrapper before any module runs.

Load-bearing gotchas

Import-free JS belongs in ui/public/, not ui/src/pages/

Vite 8 / rolldown 1.1.4 silently drops function declarations that have no callers within their own module — functions referenced only from HTML onclick handlers are treated as dead code.

Rule: any JS file with zero ES6 import statements belongs in ui/public/assets/js/iris/. Vite copies public/ verbatim; nothing is tree-shaken. JS files that import from $lib or npm packages belong in ui/src/pages/.

Every npm package referenced by a vite-plugin-static-copy target must also be declared in package.json — npm ci will not install transitive deps, and the plugin silently skips missing source paths (producing CDN 404s at runtime).

Use vite-plugin-static-copy ^3.3.0 or later for Vite 8 compatibility — 1.x and 2.x declare peer vite: "^5.0.0 || ^6.0.0" and fail to install against Vite 8.

When a file cannot move to ui/public/, expose its functions on window. case.summary.js imports crc32, so it must stay in ui/src/pages/ and the verbatim-copy escape route is unavailable. Rolldown tree-shook its onclick-only functions and the Generate-report button silently died (report_template_selector is not defined). The fix is an explicit window assignment at the end of the file:

Object.assign(window, {
    report_template_selector, act_report_template_selector,
    gen_report, gen_act_report,
});

The assignment is itself an in-module use, so rolldown keeps the functions, and it makes them the globals the inline handlers resolve against. Note that rollupOptions.treeshake: false does not save uncalled declarations under rolldown — do not rely on it.

Jinja {% inside <script> blocks

Jinja processes {% delimiters everywhere in the template file, including inside <script> tags and /* */ JS comments. Writing {% block javascripts %} in a JS comment causes TemplateSyntaxError. Write it in plain English.

jQuery load order

<script> blocks in {% block content %} run at browser parse time — before {% block javascripts %} where jQuery loads. Calling $() directly at IIFE top level throws ReferenceError: $ is not defined. Wrap jQuery wiring in:

document.addEventListener('DOMContentLoaded', function() {
  // $ is available here
});

Vanilla JS (document.getElementById, fetch, MutationObserver) is safe at IIFE top level.

Two navigation templates

The header exists twice, and both are live:

Template Used by
includes/navigation.html Dashboard, manage and admin pages
includes/navigation_ext.html Case pages (timeline, IOCs, notes, assets, tasks, evidence, graph)

Adding a header element to only one makes it vanish across half the application — and the case pages are where analysts spend most of their time, so testing on the dashboard alone will not catch it. Put shared header UI in a partial and include it from both, as case_notifications_bell.html does.

Both navigation templates render before jQuery loads, so any behaviour attached there must be vanilla JS wired from DOMContentLoaded (see jQuery load order).

CSRF on multipart requests

The JSON-body CSRF rule extends to multipart/form-data. IRIS-NG's CSRF validator reads request.form only; append csrf_token as a form field in FormData:

const body = new FormData();
body.append('file', file);
body.append('csrf_token', document.getElementById('csrf_token').value);

CSRF on pages with no WTForms form

To get a #csrf_token input on a page that has no WTForms form, pass a bare FlaskForm() to the template and render:

<form style="display:none;">{{ form.hidden_tag() }}</form>

{{ csrf_token() }} is not a Jinja global in this app — it renders empty.

.data() vs .attr() for numeric-looking data-* values

jQuery .data('cluster-id') auto-casts numeric-looking strings to integers. Use .attr('data-cluster-id') when the value needs to stay a string for === comparison against server-returned strings.

Inline onclick with user values

Never embed user-controlled values inside onclick="fn('...')" string literals. Any .replace() applied after HTML-escaping misses backslashes and other metacharacters. Use data-* attributes + a delegated handler instead:

<button data-case-id="{{ case.case_id }}">Click</button>
document.addEventListener('click', function(e) {
  const btn = e.target.closest('[data-case-id]');
  if (!btn) return;
  const id = btn.getAttribute('data-case-id');  // .attr(), not .data()
});

String .replace() in normalization/sanitization

str.replace('x', 'y') only replaces the first occurrence. Use /regex/g:

phaseName.replace(/&/g, 'and')

label for= must match input id= exactly

Mismatches break autofill, a11y, and click-to-focus. Always set for= to the input's id=, not a descriptive name.

filterXSS / sanitizeHTML is not an attribute escape

filterXSS sanitizes HTML tag/attribute names — it does not escape " inside attribute values. Use irisAttrEscape() (defined in suggesttag.js) when interpolating values into data-val="..." or any HTML attribute.

Chart.js in a hidden Bootstrap tab

Defer chart rendering to shown.bs.tab — hidden tab panes have offsetWidth = 0 and Chart.js draws all elements at (0,0).

Chart.js responsive + flip-card

In a Bootstrap flip-card back face, defer the render() call 50 ms after showing the chart slot (CSS transition from position:absolute to normal flow takes one frame).

Alembic begin_transaction()

The begin_transaction() wrapper in source/app/alembic/env.py was commented out upstream, causing all ALTER TABLE migrations to silently fail. iris-ng restores it. Watch for accidental reverts on upstream cherry-picks.

ORM CHECK constraints belong on the model, not just the migration

db.create_all() runs before Alembic. If a CHECK constraint is only in the migration file, _has_table guard skips op.create_table on fresh stacks and the constraint never lands. Always define CHECK + UNIQUE constraints on the ORM model's __table_args__.

Never hash wall-clock values into a content cache key

input_hash should include only what changes when the data changes, not when the clock advances. A field like hours_since_last_activity (derived from utcnow()) changes every call — hash the underlying updated_at timestamps instead.

Thread pool workers need their own Flask app context

Workers in a ThreadPoolExecutor do not inherit the calling thread's app context:

def _worker(app, ...):
    with app.app_context():
        ...  # ORM queries here

Return primitive IDs from workers; re-fetch ORM objects in the main thread after the pool completes.

Celery fork-safety = NullPool for workers

Configured in source/app/__init__.py when "worker" in sys.argv. Do not add per-module db.session.remove() shims — they detach ORM objects passed in by task_hook_wrapper.

Seed catalogs need slug-keyed upserts, not create_safe

create_safe matches on ALL kwargs; a name or description edit creates a duplicate. Slug/key-keyed upsert:

existing = db.session.query(Skill).filter_by(slug=slug).first()
if not existing:
    db.session.add(Skill(slug=slug, ...))

User.user is the login field — there is no User.user_login

The User ORM model stores the login name in .user, not .user_login. Using .user_login raises AttributeError at runtime (gunicorn unhandled exception → HTTP 500 before the endpoint's own error handler runs).

Field Meaning
User.user Login name (the value used to sign in)
User.name Display name

This has caused two separate HTTP 500 regressions (enqueue_ai_job and suggest_analysts_for_case). Any code that builds user-display strings from a User ORM object must use u.user for login and u.name for display name.

Module config schemas are frozen at first registration

IRIS caches a module's configuration schema in iris_module.module_config when the module is first registered. register_module() returns early with "Module already exists in Iris" on every later boot and never touches it again.

So adding a parameter to a module's module_configuration has no effect on an existing install: it never appears in the admin form. Worse, is_mod_configured() reads config['mandatory'] unguarded, so any incomplete entry raises KeyError → HTTP 500 on /manage/modules/list → an opaque "DataTables warning … Ajax error" alert with no clue as to the cause.

reconcile_module_configurations() in post_init.py now runs on every boot for all registered modules and fixes the drift: parameters declared in code but missing from the DB are added with their default, removed ones are dropped, metadata is refreshed, and the admin-set value is always preserved. It runs even when prevent_post_mod_repush is set, since it only refreshes already-registered modules and never re-adds a deleted one.

You therefore do not need to do anything special when adding a config parameter — but if you ever hand-edit module_config in the database, write a complete entry (param_name, param_human_name, type, default, mandatory, section), not just param_name + value.

MISP analyst-data API returns a misleading 403

Two shapes MISP does not document clearly, both established against a live 2.5.34 instance. Each returns 403 "Could not add analyst_data", which reads like a permissions failure and is not:

  • The body must be wrapped: POST analyst_data/add/Note requires {"Note": {...}}. A flat body is rejected.
  • distribution must be 0–3. Unlike attributes and event reports, analyst data rejects 5 ("inherit event"), so notes cannot mirror their event and the level has to be resolved by the caller.

Prose belongs in an Event Report (eventReports/add/<event_id>), which renders Markdown, rather than in a type=comment attribute.

response_api_error() cannot set a status code

Its signature is response_api_error(message, data=None) and it always returns HTTP 400. The second positional argument is data, not a status — so

return response_api_error("Backend unavailable", 503)   # emits 400 with data: 503

Several existing calls in the correlation blueprint do exactly this. When you need a status other than 400, build the body via response() directly, keeping the same shape:

from app.blueprints.responses import response

return response(409, data={'message': '...', 'data': {'reason': 'manual_edit_present'}})

v2 API response shape has no wrapper

/api/v2/ returns the payload directly — no {status, message, data} envelope. Reading resp.data on a v2 response returns undefined.

Marshmallow Integer fields reject empty strings

fields.Integer(allow_none=True) accepts None and valid integers but raises ValidationError on "". Jinja renders nullable integer fields as value="{{ x or '' }}" (empty string when None), and serializeObject() serializes those as "".

Convert empty strings to null in the JS before JSON.stringify:

['retention_months', 'capacity_planning_window_months', 'some_other_int'].forEach(function(k) {
    if (data_sent[k] === '') { data_sent[k] = null; }
});
post_request_api('/manage/settings/update', JSON.stringify(data_sent), true);

This pattern applies to any settings form that has optional numeric fields rendered via value="{{ settings.x or '' }}" in the template.

Jinja guard for shared add/edit modals

Modals shared between add (user=None) and edit (user=<dict>) paths: {% if user and user.some_field is not none %} — not just {% if user.some_field ... %}. x.attr is not none still evaluates x.attr and raises UndefinedError when x is None.

A green docker build does not validate a setuptools bump

setuptools >= 81 removed pkg_resources. Two pinned transitive dependencies still import it at module load on live application code paths:

Package Import site Why it cannot simply be bumped
docxcompose properties.py (bundled template read) The vendored docx_generator-0.8.0 wheel hard-pins docxcompose==1.1.2 (plus python-docx==1.1.2, docxtpl==0.19.0) — the deliberately frozen reporter stack
graphene-sqlalchemy utils.py (installed-version checks) —

The trap is that pip install succeeds and even import setuptools succeeds. pkg_resources is a separate module, so its absence only surfaces when the app boots and gunicorn crash-loops with ModuleNotFoundError: No module named 'pkg_resources'.

IRIS-NG resolves this without changing any dependency version: source/patches/depatch_pkg_resources.py runs in the Dockerfile compile-image stage after pip install and before the venv is copied to the final stage, rewriting both imports to importlib.resources / importlib.metadata. The patch is content-matched, idempotent, and fail-loud — it asserts every target it expects, so a future dependency bump that moves the code fails the build instead of silently regressing.

Acceptance bar for any setuptools major bump — "the image builds" is not enough:

  1. app, worker and ai_worker all boot healthy
  2. Generate a .docx report (exercises the docxcompose path)
  3. Build and query the GraphQL schema (exercises the graphene-sqlalchemy path)

If a new dependency starts importing pkg_resources, add a target to depatch_pkg_resources.py. Imports inside module docstrings (click_plugins), CLI-only entry points (qrcode) and test apps (werkzeug) are not on the application path and need no patch.

Adding new features

New Alembic migration

docker exec iriswebapp_app alembic -c /iriswebapp/app/alembic.ini revision \
  --autogenerate -m "add_my_feature"

Review the generated file — autogenerate doesn't catch everything. Always define CHECK + UNIQUE constraints on the model's __table_args__ (not just the migration).

New /api/v2/ blueprint

  1. Create source/app/blueprints/rest/v2/<name>/__init__.py
  2. Register the blueprint in source/app/__init__.py
  3. Follow the three-layer rule: blueprint → business → datamgmt
  4. Return data directly (no response_success() wrapper)
  5. Require CSRF on POST/PUT/DELETE
  6. Check access control with the existing ac_api_requires() decorator

New AI orchestrator

See AI Features → Adding a new AI orchestrator.

New working-timeline ingest source

See Dual Timeline → Adding a new ingest source.

New M2M relationship (inverse-chip pattern)

The inverse-chip pattern (violet pill chips in modal B linking back to modal A) is established across IOC↔Note and Asset↔Evidence. Chip styling:

  • Background: rgba(139, 92, 246, 0.12)
  • Border: rgba(139, 92, 246, 0.35)
  • Text: #d4c4ff
  • Deep-link: ?shared=<id> auto-opens the linked modal

GitHub Actions

Every workflow file needs an explicit permissions: block immediately after on::

permissions:
  contents: read

Add job-level permissions: for any job needing additional scope — don't promote to workflow level.

Security patterns

Rule File
Path injection: use allowlist dict, not f-string is_safe_url
Tag search: POST /tags/index not GET /tags/search/<term> misp_sync_client.py
Never log secret[:N] — log length instead seed_2025_test_case.py
Strip config sub-keys from printed output iris_misp_sync_dev.py
Open-redirect: allowlist paths starting with /, no scheme, no netloc, no // is_safe_url

Contributing

IRIS-NG is the Community Edition — community-maintained and LGPL-3.0. Contributions are welcome.

Branches

Branch Role
main The active development branch. New work lands here and releases are tagged from it. Pull requests target main.
upstream-fixes Created lazily if upstream DFIR-IRIS ships a bugfix worth cherry-picking

An older develop branch exists in some clones but is retired — it has unrelated history to main (no common ancestor) and should not be used as a PR base.

Before opening a PR

  1. Check the roadmap and open issues — the work may already be tracked.
  2. Follow CODESTYLE.md: [ADD] / [FIX] / [IMP] / [DEL] commit prefixes, f-strings only, one import per line, module-prefixed function names (iocs_create).
  3. Respect the three-layer rule — blueprints → business → datamgmt. Cross-layer imports are forbidden; a blueprint must never from app.datamgmt....
  4. Schema changes ship an Alembic migration and define any CHECK constraint on the ORM model's __table_args__ (see the gotcha above — db.create_all() runs first).
  5. Confirm the change actually deploys: check the dev-loop table at the top of this page for whether your file needs a restart, a docker cp, or a full rebuild.

Security issues

Do not open a public issue. Follow SECURITY.md.

patreon.com/zach115th
buymeacoffee.com/zach115th

Clone this wiki locally