feat(backup): expand google backup coverage

Author: steipete

CHANGELOG.md

@@ -5,6 +5,7 @@
 ### Added
 - Backup: add `gog backup` with age-encrypted Git shards, Gmail labels/raw message export, Calendar/Contacts/Tasks/Drive metadata adapters, manifest status, full decrypt-and-verify, shard `cat`, local plaintext export, docs, and security-focused regression coverage.
 - Backup: expand `gog backup push --services all` with Drive content export/download, Gmail settings, native Workspace Docs/Sheets/Slides/Form data, Apps Script projects, Chat, Classroom, best-effort optional service error shards, and plaintext Drive file export.
+- Backup: extend `--services all` with Drive permissions/comments/revisions, Calendar ACL/settings/colors, contact groups, Cloud Identity groups, Workspace Admin Directory users/groups/members, Keep notes, and local Gmail message caching for resumable full-mailbox fetches.
 
 ### Fixed
 - Gmail: auto-fill draft reply subjects from the original message when `gmail drafts create --reply-to-message-id` omits `--subject`. (#488) — thanks @jbowerbir.

README.md

@@ -747,10 +747,13 @@ private age identity locally at `~/.gog/age.key`; GitHub only receives public
 The private `AGE-SECRET-KEY-...` value must stay local or in a password manager.
 
 Supported backup services are `gmail`, `gmail-settings`, `calendar`,
-`contacts`, `tasks`, `drive`, `workspace`, `appscript`, `chat`, and
-`classroom`; `all` expands to those services. Drive now stores metadata plus
-exported Google-native file content by default. Non-Google binary Drive files
-are metadata-only unless `--drive-binary-contents` is set. Workspace inventories
+`contacts`, `tasks`, `drive`, `workspace`, `appscript`, `chat`, `classroom`,
+`groups`, `admin`, and `keep`; `all` expands to those services. Drive stores
+metadata, permissions, comments, revisions, and exported Google-native file
+content by default. Non-Google binary Drive files are metadata-only unless
+`--drive-binary-contents` is set. Gmail raw-message fetches use a local cache by
+default so interrupted full-mailbox backups can resume; use
+`--gmail-refresh-cache` to force a refetch. Workspace inventories
 Docs/Sheets/Slides and backs up Forms/responses discovered through Drive; add
 `--workspace-native` for full native Docs/Sheets/Slides API JSON.
 Optional Workspace-only services use `--best-effort` by default, recording

docs/backup.md

@@ -71,18 +71,23 @@ remote.
 
 Supported services:
 
-- `gmail`: labels and raw MIME messages.
+- `gmail`: labels and raw MIME messages. Fetched raw messages are cached under
+  the local user cache by default so interrupted full-mailbox runs can resume
+  the expensive message download phase; use `--no-gmail-cache` or
+  `--gmail-refresh-cache` to bypass it.
 - `gmail-settings`: filters, forwarding addresses, auto-forwarding, send-as
   aliases, vacation responder, delegate visibility, POP, IMAP, and language
   settings.
-- `calendar`: calendar list entries and all events, including deleted events.
-- `contacts`: People API contacts and other contacts.
+- `calendar`: calendar list entries, ACL rules, Calendar settings/colors, and
+  all events, including deleted events.
+- `contacts`: People API contacts, other contacts, and contact groups.
 - `tasks`: task lists and tasks, including completed, deleted, hidden, and
   assigned tasks.
-- `drive`: shared drives, Drive file metadata, and downloaded/exported file
-  content. Google Docs export as `.docx` and Markdown, Sheets as `.xlsx`,
-  Slides as `.pptx` and PDF, Drawings as PNG and PDF, and binary files as
-  metadata-only unless `--drive-binary-contents` is set.
+- `drive`: shared drives, Drive file metadata, permissions, comments, revision
+  metadata, and downloaded/exported file content. Google Docs export as `.docx`
+  and Markdown, Sheets as `.xlsx`, Slides as `.pptx` and PDF, Drawings as PNG
+  and PDF, and binary files as metadata-only unless `--drive-binary-contents`
+  is set.
 - `workspace`: Docs/Sheets/Slides inventory plus Forms and form responses
   discovered through Drive. Add `--workspace-native` to fetch full native
   Docs/Sheets/Slides API JSON.
@@ -92,21 +97,31 @@ Supported services:
   access.
 - `classroom`: courses, topics, announcements, coursework, materials, and
   submissions visible to the authenticated account.
+- `groups`: Cloud Identity groups the account belongs to, plus member lists
+  when the API permits them.
+- `admin`: Workspace Admin Directory users, groups, and group members. This is
+  Workspace-only and requires the existing Admin SDK/domain-wide delegation
+  setup.
+- `keep`: Google Keep notes. This is Workspace-only and requires the existing
+  Keep service-account setup.
 
 `all` expands to every supported service. Pushing a subset updates that subset
 and preserves existing shards for services that were not selected, as long as
 the age recipients are unchanged.
 
-`gog backup push` enables `--drive-contents` and `--best-effort` by default.
-Use `--no-drive-contents` for metadata-only Drive runs, or
+`gog backup push` enables `--drive-contents`, `--drive-collaboration`,
+`--gmail-cache`, and `--best-effort` by default. Use `--no-drive-contents` for
+metadata-only Drive runs, `--no-drive-collaboration` to skip per-file Drive
+permissions/comments/revisions, or
 `--drive-content-max-bytes <bytes>` to skip individual large Drive downloads.
 Drive content exports Google-native files by default; set
 `--drive-binary-contents` only when you intentionally want non-Google binary
 file bytes in Git shards. Use `--workspace-native` only when you want the
 heavier native API JSON in addition to readable Drive exports;
 `--workspace-max-files` bounds that native fetch per file type for smoke tests.
 Best-effort optional services record encrypted `errors` shards and let the rest
-of the backup finish.
+of the backup finish. The Gmail cache is only a local acceleration/resume cache;
+encrypted backup shards remain the source of truth once a push completes.
 
 ## Files
 
@@ -127,6 +142,9 @@ data/gmail/<account-hash>/messages/YYYY/MM/part-0001.jsonl.gz.age
 data/calendar/<account-hash>/...
 data/contacts/<account-hash>/...
 data/drive/<account-hash>/...
+data/groups/<account-hash>/...
+data/admin/<account-hash>/...
+data/keep/<account-hash>/...
 data/tasks/<account-hash>/...
 ```
 
@@ -238,17 +256,26 @@ Raw message payloads stay base64url encoded inside encrypted JSONL. This
 preserves the RFC 2822 message content while keeping the shard format text
 friendly.
 
+By default, each fetched raw message is also cached locally under the OS user
+cache directory (`gogcli/backup/gmail/<account-hash>/raw-v1/`). The cache stores
+the same raw message row that will be encrypted into shards and is keyed by a
+SHA-256 of the Gmail message ID, so rerunning after an interruption can reuse
+already fetched messages. `--gmail-refresh-cache` forces a refetch. The cache is
+plaintext local data; clear it if the machine should not retain local mail
+copies outside the encrypted backup/export locations.
+
 `--include-spam-trash` defaults to true. Use `--query` and `--max` for bounded
 test exports; omit them for a full mailbox scan.
 
 The Gmail settings adapter backs up account configuration through read-only
 settings endpoints. Some settings, such as delegates, can be forbidden for
 consumer accounts; those errors are kept inside the encrypted settings shard.
 
-The Calendar adapter backs up calendar list entries and all events from each
-calendar. The Contacts adapter backs up contacts and other contacts. The Tasks
-adapter backs up task lists and tasks. The Drive adapter backs up shared drives,
-file metadata, and Google-native file exports by default. Content rows store
+The Calendar adapter backs up calendar list entries, ACLs, settings, colors,
+and all events from each calendar. The Contacts adapter backs up contacts, other
+contacts, and contact groups. The Tasks adapter backs up task lists and tasks.
+The Drive adapter backs up shared drives, file metadata, permissions, comments,
+revision metadata, and Google-native file exports by default. Content rows store
 base64 bytes inside encrypted JSONL so Git only sees ciphertext; plaintext
 export decodes them back into regular files. Non-Google binary Drive bytes are
 opt-in because personal Drives can easily contain tens of gigabytes.

internal/cmd/backup.go

@@ -39,6 +39,9 @@ const (
 	backupServiceDrive         = "drive"
 	backupServiceGmail         = "gmail"
 	backupServiceGmailSettings = "gmail-settings"
+	backupServiceGroups        = "groups"
+	backupServiceAdmin         = "admin"
+	backupServiceKeep          = "keep"
 	backupServiceTasks         = "tasks"
 	backupServiceWorkspace     = "workspace"
 )
@@ -115,8 +118,11 @@ type BackupPushCmd struct {
 	DriveContents        bool   `name:"drive-contents" help:"Download/export Drive file contents into encrypted shards" default:"true" negatable:""`
 	DriveBinaryContents  bool   `name:"drive-binary-contents" help:"Include non-Google Drive binary file bytes in encrypted shards"`
 	DriveContentMaxBytes int64  `name:"drive-content-max-bytes" help:"Skip individual Drive content exports larger than this many bytes; 0 means unlimited" default:"0"`
+	DriveCollaboration   bool   `name:"drive-collaboration" help:"Back up Drive permissions, comments, and revision metadata" default:"true" negatable:""`
 	WorkspaceNative      bool   `name:"workspace-native" help:"Fetch full native Docs/Sheets/Slides API JSON in addition to Drive exports"`
 	WorkspaceMaxFiles    int    `name:"workspace-max-files" help:"Max Docs/Sheets/Slides files per type for native Workspace metadata; 0 means all" default:"0"`
+	GmailCache           bool   `name:"gmail-cache" help:"Cache fetched Gmail raw messages locally so interrupted full backups can resume" default:"true" negatable:""`
+	GmailRefreshCache    bool   `name:"gmail-refresh-cache" help:"Refetch Gmail messages even when a local backup cache entry exists"`
 	BestEffort           bool   `name:"best-effort" help:"Record optional service errors as backup rows and continue" default:"true" negatable:""`
 }
 
@@ -170,6 +176,7 @@ func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
 				IncludeContents: c.DriveContents,
 				IncludeBinary:   c.DriveBinaryContents,
 				MaxContentBytes: c.DriveContentMaxBytes,
+				IncludeCollab:   c.DriveCollaboration,
 			})
 			if err != nil {
 				return err
@@ -181,6 +188,8 @@ func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
 				Max:              c.Max,
 				IncludeSpamTrash: c.IncludeSpamTrash,
 				ShardMaxRows:     c.ShardMaxRows,
+				CacheMessages:    c.GmailCache,
+				RefreshCache:     c.GmailRefreshCache,
 			})
 			if err != nil {
 				return err
@@ -192,6 +201,30 @@ func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
 				return err
 			}
 			snapshots = append(snapshots, snapshot)
+		case backupServiceGroups:
+			snapshot, err := c.buildOptionalSnapshot(flags, backupServiceGroups, func() (backup.Snapshot, error) {
+				return buildGroupsBackupSnapshot(ctx, flags, c.ShardMaxRows)
+			})
+			if err != nil {
+				return err
+			}
+			snapshots = append(snapshots, snapshot)
+		case backupServiceAdmin:
+			snapshot, err := c.buildOptionalSnapshot(flags, backupServiceAdmin, func() (backup.Snapshot, error) {
+				return buildAdminBackupSnapshot(ctx, flags, c.ShardMaxRows)
+			})
+			if err != nil {
+				return err
+			}
+			snapshots = append(snapshots, snapshot)
+		case backupServiceKeep:
+			snapshot, err := c.buildOptionalSnapshot(flags, backupServiceKeep, func() (backup.Snapshot, error) {
+				return buildKeepBackupSnapshot(ctx, flags, c.ShardMaxRows)
+			})
+			if err != nil {
+				return err
+			}
+			snapshots = append(snapshots, snapshot)
 		case backupServiceTasks:
 			snapshot, err := buildTasksBackupSnapshot(ctx, flags, c.ShardMaxRows)
 			if err != nil {
@@ -211,7 +244,7 @@ func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
 			}
 			snapshots = append(snapshots, snapshot)
 		default:
-			return fmt.Errorf("unsupported backup service %q (supported: all, appscript, calendar, chat, classroom, contacts, drive, gmail, gmail-settings, tasks, workspace)", service)
+			return fmt.Errorf("unsupported backup service %q (supported: all, admin, appscript, calendar, chat, classroom, contacts, drive, gmail, gmail-settings, groups, keep, tasks, workspace)", service)
 		}
 	}
 	result, err := backup.PushSnapshot(ctx, mergeBackupSnapshots(snapshots...), c.options())
@@ -239,6 +272,8 @@ type BackupGmailPushCmd struct {
 	Max              int64  `name:"max" aliases:"limit" help:"Max Gmail messages to export; 0 means all" default:"0"`
 	IncludeSpamTrash bool   `name:"include-spam-trash" help:"Include spam and trash" default:"true"`
 	ShardMaxRows     int    `name:"shard-max-rows" help:"Max messages per encrypted shard" default:"1000"`
+	CacheMessages    bool   `name:"gmail-cache" help:"Cache fetched raw messages locally so interrupted full backups can resume" default:"true" negatable:""`
+	RefreshCache     bool   `name:"gmail-refresh-cache" help:"Refetch messages even when a local backup cache entry exists"`
 }
 
 func (c *BackupGmailPushCmd) Run(ctx context.Context, flags *RootFlags) error {
@@ -247,6 +282,8 @@ func (c *BackupGmailPushCmd) Run(ctx context.Context, flags *RootFlags) error {
 		Max:              c.Max,
 		IncludeSpamTrash: c.IncludeSpamTrash,
 		ShardMaxRows:     c.ShardMaxRows,
+		CacheMessages:    c.CacheMessages,
+		RefreshCache:     c.RefreshCache,
 	})
 	if err != nil {
 		return err