Sync dev to main #127
Merged
Sync dev to main #127
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
tis24dev
commented
Feb 2, 2026
tis24dev
commented
- Handle context deadline in input functions and add tests
- refactor: Implement selective feature-based exclusions for config snapshots
- feat: Apply exclusion patterns to archive packing and metadata
- Improve update check logging in main.go
- Improve log line classification and add debug filtering
- Expand restore categories with staged apply for access control
- Improve error handling in backup and restore workflows
- refactor: split metrics, add storage inventory and fix test resource cleanup
- Reorganize backup categories and archive layout; centralize diagnostics
- ** Extend PBS restore coverage (host/integrations, tape, S3) and align categories/docs Description**
- add PBS mount guards and fstab merge ordering
- Improve fstab restore with stable device remapping
- Document PBS mount guard and smart fstab restore
- Enable restore of PBS proxy and SSL configuration
- Add staged apply for PVE configs and mount guards
- Add PVE storage mount guard logic and improve parsing
- Refactor PVE access control restore to 1:1 file apply
- Enhance PBS access control restore with root safety rail
- Improve TFA/WebAuthn restore guidance and workflow
- Add PVE firewall restore with rollback and glob support
- Add transactional restore support for PVE HA configuration
- fix
- fix minor
- Add PVE SDN restore handling and tests
- PVE SAFE-apply: mappings, pools and ACL rollback
- Add HTTP helpers and robust release tag fetch
- deps(deps): bump github.com/gdamore/tcell/v2 from 2.13.5 to 2.13.8 in the security-patches group (deps(deps): bump github.com/gdamore/tcell/v2 from 2.13.5 to 2.13.8 in the security-patches group #122)
- deps(deps): bump golang.org/x/term from 0.38.0 to 0.39.0 (deps(deps): bump golang.org/x/term from 0.38.0 to 0.39.0 #125)
- deps(deps): bump golang.org/x/crypto from 0.46.0 to 0.47.0 (deps(deps): bump golang.org/x/crypto from 0.46.0 to 0.47.0 #123)
- deps(deps): bump golang.org/x/text from 0.32.0 to 0.33.0 (deps(deps): bump golang.org/x/text from 0.32.0 to 0.33.0 #124)
- Refine unescapeProcPath and add tests
- Handle existing PVE pools and ensure comments
- Redact secrets when applying PVE notifications
- Refactor guard cleanup and add tests
Updated ReadLineWithContext and ReadPasswordWithContext to return context.DeadlineExceeded on context deadline, instead of ErrInputAborted. Added corresponding unit tests for deadline handling in input and orchestrator prompt functions.
…pshots * Added FilesNotFound and FilesSkipped counters to CollectionStats for tracking collection outcomes * Introduced withTemporaryExcludes helper method to apply feature-based exclusions during directory snapshots without affecting other collectors * Refactored PBS config collection to exclude disabled feature files (datastore.cfg, user.cfg, remote.cfg, sync.cfg, verification.cfg, tape.cfg, network.cfg, prune.cfg) from /etc/proxmox-backup snapshot * Added collectPBSConfigFile method with detailed logging for individual PBS configuration file collection status (disabled/not-found/failed/collected) * Refactored PVE directory collection to exclude qemu-server and lxc dirs when BACKUP_VM_CONFIGS=false * Added exclusion of firewall and host.fw when BACKUP_PVE_FIREWALL=false in PVE collection * Excluded user.cfg, acl.cfg, and domains.cfg from PVE snapshot when BACKUP_PVE_ACL=false * Excluded jobs.cfg and vzdump.cron from PVE snapshot when BACKUP_PVE_JOBS=false * Wrapped PVE cluster database capture (config.db) in BACKUP_CLUSTER_CONFIG flag check * Wrapped corosync.conf exclusion in BACKUP_CLUSTER_CONFIG check for PVE snapshot * Excluded .ssh directories from root and user home collections when BACKUP_SSH_KEYS=false * Updated RCLONE_TIMEOUT_CONNECTION documentation to clarify per-command timeout behavior during cloud scanning (timer resets per rclone lsf/cat operation) * Added test cases for PBS and PVE config file exclusions based on disabled feature flags * Added test case for SSH key exclusion from root home directory when BACKUP_SSH_KEYS=false
* Introduced FindExcludeMatch function that checks patterns against both host and archive paths, enabling pattern matching with or without leading slashes * Added ExcludePatterns field to Archiver and ArchiverConfig to filter files during tar packing (checks in addToTar before adding to archive) * Updated uniqueCandidates to generate path variants for systemRootPrefix and tempDir, supporting proper pattern matching across different path representations * Modified safeCopyFile and safeCopyDir to check both source and destination paths against exclusions, skipping matched paths with FilesSkipped counter * Extended safeCmdOutput and related functions to exclude command output paths, delegating to new writeReportFile method * Refactored writeReportFile to centralize report writing with exclusion checks before filesystem operations * Updated WriteManifest to use writeReportFile for exclusion-aware manifest writing * Added PBS collector methods to check exclusions in collectPBSConfigFile and collectDatastoreNamespaces * Implemented populatePVEManifest to track PVE config status including StatusSkipped for excluded paths * Modified writeBackupMetadata to check exclusions and skip metadata generation if excluded by patterns
Enhanced logging in the update check process to provide clearer debug and info messages. Changed pre-backup check log level from Debug to Info for better visibility.
Refactored log_parser.go to better distinguish between log formats and to ignore lines with DEBUG level, even if they contain error keywords. Added helper functions for bracketed issue lines and token splitting. Updated tests to cover debug lines containing error text.
* Added `pve_notifications` category covering datacenter notification targets and matchers applied via `pvesh` API calls * Added `pve_access_control` category for users, roles, groups, ACLs, realms and secrets with staged API apply (passwords/tokens regenerated) * Added `pbs_remotes` category for remote definitions used in sync/verify jobs with staged apply capability * Added `pbs_notifications` category for PBS notification targets and matchers with staged apply * Added `pbs_access_control` category for PBS users, realms, ACLs, and secrets with staged apply (7 config files tracked) * Added `maintenance_pbs` category for PBS maintenance settings (separate from full export) * Added `filesystem` category for /etc/fstab mount point configuration with critical boot warning * Updated `pbs_config` to export-only mode, clarifying that full /etc/proxmox-backup is never written (subsets applied via new staged categories) * Extended `isStagedCategoryID` to recognize 8 total staged categories including new access control and notification categories * Added `maybeApplyAccessControlFromStage` and `maybeApplyNotificationsFromStage` calls to restore workflow UI for staged apply * Added `applyPBSRemoteCfgFromStage` function to apply remote.cfg from stage directory during PBS restoration * Extended restore mode documentation to explain three handling types (Normal/Staged/Export-only) with specific examples * Added PVE SAFE-mode secrets documentation showing regenerated passwords/tokens stored in `/tmp/proxsave/restore-stage-*/pve_access_control_secrets.json` * Clarified export-only `pve_config_export` and `pbs_config` categories are extracted separately for manual review, not applied to system * Updated restoration pass documentation to include Pass 3 for staged category apply with safe file/API apply steps
Adds detailed logging and error messages for manifest/metadata read errors during rclone cloud backup discovery, including guidance for slow remotes and rclone failures. Enhances restore workflow UI to handle unexpected EOF on input, warning users about possible TTY loss and suggesting re-running in an interactive shell.
…cleanup
- PBS Collector Refactoring: Moved datastore management and PXAR metadata logic from collector_pbs.go to the new collector_pbs_datastore.go for modularity.
- Datastore Inventory Feature: Added collector_pbs_datastore_inventory.go to collect advanced storage configurations (LVM, ZFS, Multipath, iSCSI), critical files (crypttab, multipath.conf), and referenced keys.
- System Collector: Explicitly included /etc/crypttab in the critical file collection.
- Test Suite Hygiene: Refactored numerous integration tests in internal/orchestrator that use NewFakeFS.
- Resource Cleanup: Added explicit t.Cleanup(os.RemoveAll) calls in tests to ensure removal of temporary directories ("fake folders") created during execution, preventing disk space leaks.
- Centralize all ProxSave-generated diagnostics under var/lib/proxsave-info/ (commands/inventory/runtime snapshots) - Remove legacy scattered output directories (commands/, datastores/, users/) from collectors - Fix home restore mapping by storing /home/* under ./home/<user> (avoid restoring into /users/...) - Rework category set: expand network, ssl, services; add storage_stack, user_data, proxsave_info (export-only) - Add glob support for category path matching (e.g. auto.*) - Keep backward compatibility for legacy backups (resolv.conf repair + NIC inventory fallbacks) - Update docs and align CLI/TUI restore mode labels with the new category layout
…categories/docs Description - Add new staged PBS categories: pbs_host (node/ACME/metrics/traffic-control) and pbs_tape (tape configs + encryption keys) - Expand datastore_pbs to include s3.cfg and apply it during staged PBS restore - Include proxy.key in ssl category - Update PBS STORAGE mode to include pbs_remotes (sync jobs depend on remotes) - Update PBS collector to explicitly capture the new config files and exclude all tape-related files when disabled - Update restore docs/roadmap and adjust tests accordingly
- Add CleanupGuards boolean field to Args struct for guard cleanup flag - Implement --cleanup-guards CLI flag with description for bind mount cleanup - Add validation in main run() that --cleanup-guards incompatible with 9 conflicting flags - Call orchestrator.CleanupMountGuards() when cleanup-guards mode activated - Update TestParseDefaults to verify CleanupGuards field defaults to false - Add --cleanup-guards to TestParseCustomFlags test case array - Update CleanupGuards assertion in TestParseCustomFlags test check - Modify shouldApplyPBSDatastoreBlock to log warning instead of skipping missing mounts - Add conditional logger.Warning() call when datastore mount resolves to root filesystem - Create new TestShouldApplyPBSDatastoreBlock_AllowsMountLikePathsOnRootFS test function - Reorder restore workflow to run fstab merge BEFORE PBS datastore config application - Add maybeApplyPBSDatastoreMountGuards() call before PBS config application in staging - Move entire needsFilesystemRestore block earlier in workflow lifecycle
Enhances the restore process to remap unstable /dev/* device references in fstab to stable UUID/PARTUUID/LABEL using collected inventory (blkid, lsblk JSON, PBS inventory). Updates backup collection to include block device inventory, improves SmartMergeFstab logic, and adds tests for device remapping. Also improves mount guard logic and user feedback in the restore UI.
Expanded documentation to cover PBS datastore mount guard behavior, including temporary guards for offline storage and manual cleanup via --cleanup-guards. Updated smart /etc/fstab merge details to include device remapping and normalization of entries. Clarified restore workflow and safety defaults in guides and technical docs.
Updated category definitions, staged apply logic, and documentation to support restoring PBS proxy configuration (proxy.cfg) and SSL assets (proxy.pem, proxy.key, ssl/). Adjusted restore logic to allow these files to be restored instead of always skipping them, and updated tests accordingly.
Implements staged application of PVE storage, datacenter, and backup job configurations via API after restore, with logic to skip redundant steps in cluster recovery mode. Adds mount guard logic for both PBS and PVE to protect storage mountpoints when offline, including new helpers and tests. Updates documentation to clarify staged apply behavior and category descriptions.
Introduces logic to apply mount guards for PVE storage definitions, including network and dir-backed storages, to prevent accidental writes when storage is offline during restore. Updates documentation to describe the new guard strategy, refactors guard candidate selection, and improves storage.cfg block parsing to support legacy and current formats. Adds and updates tests to cover new guard logic and parsing behavior.
Reworks the PVE access control restore logic to apply user, domain, token, shadow, and TFA config files directly to pmxcfs for standalone restores, preserving root@pam from the fresh install and ensuring root@pam retains Administrator on '/'. Updates documentation to clarify 1:1 restore behavior and cluster safety rails. Removes legacy SAFE-mode API-based secrets regeneration and related reporting.
Refactors PBS access control restoration to apply all access control and secret files 1:1 from backup, except for root@pam and its tokens/TFA, which are preserved from the fresh install for safety. Updates ACL handling to ensure root@pam retains Admin on /, and adds comprehensive tests for the new behavior. Documentation is updated to clarify the new 1:1 restore logic and root safety measures.
Enhances documentation, logging, and UI to provide clearer guidance for restoring TFA/WebAuthn, especially regarding the importance of restoring 'network' and 'ssl' for maximum compatibility. Adds logic to recommend and optionally auto-select these categories in custom restore mode, warns about hostname/origin changes, and introduces helper/test code for these features.
Introduces support for restoring PVE firewall configuration with a transactional rollback timer, including a dedicated firewall rollback backup. Adds glob path expansion to safety backup logic, updates category definitions and documentation, and provides comprehensive tests for new backup and restore behaviors.
Introduces a new 'pve_ha' category for PVE High Availability configuration, including staged restore logic, rollback timer, and user confirmation workflow. Updates documentation to reflect the new category, implements backup/restore safety for HA config, and adds tests for HA restore behavior. Integrates HA rollback backup creation and transactional apply into the main restore workflow.
Add support for restoring PVE SDN definitions from staging: introduce a new pve_sdn category, implement staged apply logic (applyPVESDNFromStage + maybeApplyPVESDNFromStage) and integrate it into the restore UI workflow. Includes unit tests for the SDN apply behavior, updates to staging category checks, and documentation updates (RESTORE_GUIDE.md and RESTORE_DIAGRAMS.md). Also updates internal .backup.lock timestamp.
Add cluster SAFE-apply support for PVE resource mappings and pools, plus an access-control rollback workflow. Collector now exports pve resource pools and cluster mapping JSON so SAFE restores can reapply /cluster/mapping/* and pool definitions. New orchestrator modules implement applying mappings (pvesh) and pools (pveum), parsing exported data and merging with existing state. Add access-control UI flow that can arm a timed rollback (writes marker/script/log, uses systemd-run or background timer) and create an access-control-only rollback backup. Wire the new flows into the restore workflow, extract SAFE inventory when needed, and add tests and documentation updates describing the SAFE apply behavior.
Introduce fetch() and download() helpers that use curl or wget to centralize HTTP operations and error handling. Use fetch to retrieve the GitHub releases JSON, prefer jq to extract .tag_name with a regex fallback if jq is unavailable. Remove the previous duplicate download function, adjust step numbering/comments, and keep checksum verification, extraction and install flow intact. These changes improve portability and make release tag detection more reliable.
… the security-patches group (#122) deps(deps): bump github.com/gdamore/tcell/v2 Bumps the security-patches group with 1 update: [github.com/gdamore/tcell/v2](https://github.com/gdamore/tcell). Updates `github.com/gdamore/tcell/v2` from 2.13.5 to 2.13.8 - [Release notes](https://github.com/gdamore/tcell/releases) - [Changelog](https://github.com/gdamore/tcell/blob/main/CHANGESv3.md) - [Commits](gdamore/tcell@v2.13.5...v2.13.8) --- updated-dependencies: - dependency-name: github.com/gdamore/tcell/v2 dependency-version: 2.13.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: security-patches ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.38.0 to 0.39.0. - [Commits](golang/term@v0.38.0...v0.39.0) --- updated-dependencies: - dependency-name: golang.org/x/term dependency-version: 0.39.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.46.0 to 0.47.0. - [Commits](golang/crypto@v0.46.0...v0.47.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.47.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.32.0 to 0.33.0. - [Release notes](https://github.com/golang/text/releases) - [Commits](golang/text@v0.32.0...v0.33.0) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-version: 0.33.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Replace strconv-based octal parsing with a lightweight manual decode in unescapeProcPath and add unit tests. The function now computes octal values with bit shifts and preserves escaped sequences whose value is out of the byte range (>255) or malformed, matching /proc/self/mountinfo behavior. Also remove the now-unused strconv import and add tests covering spaces, tabs, newlines, backslashes, incomplete/non-octal sequences, out-of-range escapes, and the null byte.
Add detection of existing PVE pools and improve apply logic to avoid treating "already exists" add errors as failures. Implement listPVEPoolIDs to parse `pveum pool list` and pvePoolAlreadyExists to inspect add output. Refactor applyPVEPoolsDefinitions to prefetch existing pools, reuse a trimmed comment variable, treat adds that fail because the pool already exists as OK, update the cached set when a pool is created, and always attempt to set the comment. Add unit tests covering the case where an existing pool without a comment is treated as success and where add failures for missing pools are treated as errors. Log pool-listing failures but do not abort the overall apply flow.
Add support for redacting sensitive fields when restoring Proxmox notification endpoints and matchers. Introduce RedactFlags on proxmoxNotificationSection and collect redact flags per-section (including defaults like --password, --token, --secret, --apikey, --api-key and inferred flags from entry keys). applyPveshObject now accepts redact flags and will call runPveshSensitive when needed to avoid leaking secrets in command errors; fallback behavior for create/set preserved. Added helpers notificationRedactFlagsFromEntries and notificationRedactFlags and a test to ensure secrets are redacted. Also import strings in the test file.
Introduce injectable OS/syscall wrappers (geteuid, stat, ReadFile, RemoveAll, Unmount) to make CleanupMountGuards testable. Rework mountinfo parsing to distinguish visible vs hidden guard mounts using mount IDs, return counts, and skip unmounting guards that are hidden under other mounts. Sort and deduplicate targets, use the injected functions for file/FS operations, and avoid removing the guard directory when any guard mounts remain. Add unit tests for guardMountpointsFromMountinfo and CleanupMountGuards behaviour (visible vs hidden mounts, unmount+remove path).
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
There was a problem hiding this comment.
Pull request overview
This PR synchronizes the development branch with the main branch, incorporating a large number of enhancements focused on improving backup/restore functionality, refactoring configuration handling, expanding test coverage, and adding new features for PVE/PBS systems.
Changes:
- Enhanced restore workflow with staged configuration application, improved error handling, and expanded category coverage for PVE/PBS systems
- Refactored backup exclusion patterns and manifest tracking to provide better visibility into what was collected/skipped
- Added mount guard logic for offline storage, smart fstab merging with device remapping, and comprehensive PBS datastore inventory collection
- Improved cloud backup scanning with per-command timeouts, better error messages, and progress reporting
- Expanded test coverage across restore, backup, and orchestrator modules with proper cleanup
Reviewed changes
Copilot reviewed 95 out of 110 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| internal/orchestrator/restore_*.go | Restore workflow enhancements: staged apply for PVE/PBS configs, SDN/HA/firewall/notifications support, improved error handling |
| internal/orchestrator/pve_safe_apply_*.go | New PVE resource pools and mappings restore with API-based application |
| internal/orchestrator/guards_cleanup*.go | Mount guard cleanup logic for PBS/PVE datastores on offline mountpoints |
| internal/backup/collector*.go | Enhanced backup manifest tracking, exclusion pattern handling, PBS datastore inventory collection |
| internal/backup/archiver*.go | Archive packing now respects exclusion patterns |
| internal/orchestrator/decrypt*.go | Refactored decrypt workflow to use UI abstraction layer |
| internal/orchestrator/backup_sources.go | Cloud backup discovery with per-command timeouts and progress reporting |
| internal/orchestrator/categories.go | Expanded category definitions for PVE/PBS with new staged/export categories |
| internal/cli/args.go | Added --cleanup-guards flag |
| install.sh | Improved HTTP helpers with curl/wget fallback and robust release tag parsing |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.