feat: add admin backup and restore workflow

.env.example

@@ -184,3 +184,13 @@ BACKUP_MAX_FILE_SIZE=1073741824
 BACKUP_MAX_TOTAL_SIZE=10737418240
 BACKUP_MAX_COMPRESSION_RATIO=200
 BACKUP_MAX_MANIFEST_SIZE=10485760
+# Native Admin backup and restore are always available to administrators.
+# Operational limits only; BACKUP_TEMP_DIR is a shared base outside DATA_DIR.
+BACKUP_UPLOAD_MAX_SIZE=1073741824
+BACKUP_OPERATION_TIMEOUT=1800
+BACKUP_DOWNLOAD_TTL=600
+# BACKUP_TEMP_DIR=/tmp/webssh-backup-operations
+RATELIMIT_BACKUP_CREATE=3 per hour
+RATELIMIT_BACKUP_UPLOAD=5 per hour
+RATELIMIT_BACKUP_DOWNLOAD=10 per hour
+RATELIMIT_BACKUP_RESTORE=3 per hour

README.md

@@ -796,7 +796,81 @@ designed for a read-only snapshot and never migrates plaintext legacy keys.
 Its report omits key content, configured key names, filenames, paths, and the
 `SECRET_KEY`.
 
-### Backup, Restore, and Secret Rotation
+### Web Backup and Restore
+
+Administrators can create, download, verify, and restore backups from
+**Administration > Backup & Restore**. This native feature is always present;
+there are no feature flags that can silently disable backup or restore.
+
+Web backup uses SQLite's native backup API and briefly coordinates persistent
+file writers while it captures the database and file-based stores. WebSSH stays
+online during creation. The temporary snapshot is deleted immediately after the
+verified ZIP has been created. The verified archive is kept in a private
+directory outside `DATA_DIR` only until its one-time, session-bound download or
+until its TTL expires.
+
+The archive includes the SQLite database, application settings, user profiles,
+`known_hosts`, persisted application secret, SSH key metadata, encrypted private
+keys, and the other persistent files covered by the CLI format. Runtime `logs/`,
+`tmp/`, transient uploads, and incomplete transfer data remain excluded. New
+web and CLI archives use format version 2 and are mutually compatible. Format
+v2 records the WebSSH data-schema version, creation time, and producer in the
+manifest. Existing format-v1 CLI archives remain supported as legacy schema 0
+backups.
+
+Archive verification and restore compatibility are separate decisions. A safe,
+well-formed archive can be inspected even when it cannot be restored by the
+running version. The Admin validation result shows the archive format, backup
+and current data-schema versions, creation time, legacy status, and a
+compatibility reason. Backups with the current schema are accepted. Older
+schemas are accepted only when WebSSH has a complete registered migration path.
+Backups with a newer schema are blocked server-side before restore preparation
+and checked again before the destructive operation starts. Restoring a newer
+backup into an older WebSSH release is not supported.
+
+To restore, upload an archive in the same Admin tab. WebSSH verifies its
+manifest, checksums, sizes, members, compression limits, and format before it
+shows a non-sensitive summary. Restore then requires two explicit confirmations,
+the exact phrase `RESTORE`, and the current administrator password. The service
+enters maintenance mode, rejects new writes and SSH sessions, closes active
+runtime activity, creates an online-consistent emergency rollback archive, and
+replaces the persistent state. All browser sessions are invalidated.
+
+After a successful restore, the process terminates intentionally. Docker
+Compose and Portainer deployments using `restart: unless-stopped` restart the
+container automatically. The Admin page reports the operation while possible;
+a disconnect during the final step means the administrator should wait for
+`/ready` and sign in again. An interrupted restore is detected on startup and
+rolled back from the emergency archive. If both restore and rollback fail,
+maintenance mode remains active and the operator must use the CLI restore path.
+A successful confirmed CLI restore clears this recovery-only maintenance state;
+the following application start removes the retained temporary rollback files.
+
+Backup archives are highly sensitive. HTTPS protects transport only; it does
+not encrypt the downloaded ZIP at rest. Store downloads encrypted, off-host,
+with administrator-only access, and dispose of them according to a retention
+policy.
+
+Web restore is intentionally treated as a high-risk administrative operation,
+not as a routine user action. Keep the Admin interface behind HTTPS and trusted
+access controls, retain an encrypted off-host backup, and keep the offline CLI
+restore procedure available when the web process or its current data schema
+cannot start safely.
+
+Operational configuration:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `BACKUP_UPLOAD_MAX_SIZE` | `1073741824` | Maximum streamed web upload size in bytes |
+| `BACKUP_OPERATION_TIMEOUT` | `1800` | Operation and retained-status timeout in seconds |
+| `BACKUP_DOWNLOAD_TTL` | `600` | TTL for generated downloads and verified uploads in seconds |
+| `BACKUP_TEMP_DIR` | system temp + `webssh-backup-operations` | Private temporary base outside `DATA_DIR`; WebSSH creates an isolated namespace per resolved data directory |
+| `RATELIMIT_BACKUP_CREATE` | `3 per hour` | Per-admin/IP creation rate |
+| `RATELIMIT_BACKUP_UPLOAD` | `5 per hour` | Per-admin/IP upload rate |
+| `RATELIMIT_BACKUP_DOWNLOAD` | `10 per hour` | Per-admin/IP download rate |
+| `RATELIMIT_BACKUP_RESTORE` | `3 per hour` | Per-admin/IP restore-attempt rate |
+
+### CLI Backup, Restore, and Secret Rotation
 
 Run mutating maintenance commands only while every WebSSH application process
 that uses the data directory is stopped. Archives contain the database, user
@@ -957,9 +1031,10 @@ private, and reserved targets after DNS resolution.
 - Restrict and encrypt backups of `DATA_DIR`; they contain account metadata,
   encrypted private keys, and may include the Docker-generated `SECRET_KEY`.
   Runtime logs and incomplete transfers are excluded.
-- Stop all WebSSH processes before backup creation, restore, or secret rotation.
-  Verify archives before transferring or restoring them, and restart
-  immediately after a successful persisted-secret rotation.
+- Use the Admin workflow for an online-consistent backup. Stop all WebSSH
+  processes before CLI backup creation, CLI restore, or secret rotation. Verify
+  archives before transferring or restoring them, and restart immediately after
+  a successful persisted-secret rotation.
 - Define a retention and secure-disposal policy for `DATA_DIR/deleted_users`.
   Account deletion quarantines those files to prevent numeric user-id reuse
   from exposing them, but does not wipe them automatically.

app/__init__.py

@@ -42,6 +42,8 @@ def _initialize_persistent_storage(app):
         return
 
     config.DATA_DIR.mkdir(parents=True, exist_ok=True)
+    from .session_epoch import current_epoch
+    current_epoch()
     from .audit_logger import initialize_file_logging
     initialize_file_logging(config.DATA_DIR)
     with app.app_context():
@@ -74,6 +76,10 @@ def create_app(
         max_workers=config.BACKGROUND_WORKERS
     )
 
+    from .maintenance_mode import is_active, recover_interrupted_restore
+    if initialize_storage:
+        recover_interrupted_restore()
+
     for warning in config.SECURITY_CONFIG_WARNINGS:
         log_warning('Deployment security warning', warning=warning)
 
@@ -135,6 +141,27 @@ def hide_disabled_admin_panel():
         ):
             abort(404)
 
+    @app.before_request
+    def enforce_restore_maintenance_and_session_epoch():
+        if is_active() and request.path not in {
+            '/health',
+            '/ready',
+            '/admin/api/backups/restore/status',
+        }:
+            return jsonify({
+                'error': 'WebSSH is in restore maintenance mode',
+                'code': 'maintenance',
+            }), 503
+        if initialize_storage and current_user.is_authenticated:
+            from .session_epoch import current_epoch
+            epoch = current_epoch()
+            stored_epoch = session.get('_auth_epoch')
+            if stored_epoch is None:
+                session['_auth_epoch'] = epoch
+            elif stored_epoch != epoch:
+                logout_user()
+                session.clear()
+
     trusted_proxies = config.TRUSTED_PROXIES
     if trusted_proxies > 0:
         app.wsgi_app = ProxyFix(
@@ -160,13 +187,16 @@ def hide_disabled_admin_panel():
     app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{config.DATA_DIR / "app.db"}'
     app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
     db.init_app(app)
+    from .backup_coordination import install_sqlalchemy_coordination
+    install_sqlalchemy_coordination()
     init_auth(app)
     from .request_limits import init_request_limits
     from .webauthn_routes import webauthn_blueprint
     init_request_limits(app)
     csrf.init_app(app)
     from .cli import register_cli
     from .audit_export import audit_export_blueprint
+    from .admin_backup import admin_backup_blueprint
     from .health import health_blueprint
     from .host_key_routes import host_key_blueprint
     from .oidc_routes import init_oidc, oidc_blueprint
@@ -176,6 +206,7 @@ def hide_disabled_admin_panel():
     if initialize_oidc:
         init_oidc(app)
     app.register_blueprint(audit_export_blueprint)
+    app.register_blueprint(admin_backup_blueprint)
     app.register_blueprint(health_blueprint)
     app.register_blueprint(host_key_blueprint)
     app.register_blueprint(oidc_blueprint)
@@ -185,6 +216,12 @@ def hide_disabled_admin_panel():
     if initialize_storage:
         _initialize_persistent_storage(app)
     if start_runtime:
+        from .backup_operations import backup_operations
+        backup_operations.cleanup_orphans()
+        app.extensions['runtime_lifecycle'].start_job(
+            'backup-operation-cleanup',
+            backup_operations.cleanup_loop,
+        )
         transfer_runtime_binding = transfer_manager.bind_runtime()
         app.extensions['runtime_lifecycle'].register_shutdown_callback(
             'active_transfers',
@@ -237,6 +274,9 @@ def add_security_headers(response):
         if not config.DEBUG:
             response.headers['Strict-Transport-Security'] = 'max-age=31536000; includeSubDomains'
 
+        if initialize_storage and session.get('_user_id') is not None:
+            from .session_epoch import current_epoch
+            session['_auth_epoch'] = current_epoch()
         return response
 
     from . import socket_events, command_manager, connection_pool