Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --skip-db option to dump command #30613

Merged
merged 6 commits into from
Apr 21, 2024

Conversation

kemzeb
Copy link
Contributor

@kemzeb kemzeb commented Apr 20, 2024

Attempts to resolve #28720.


Note that I am not a Gitea administrator so I don't normally use the gitea CLI. Just saw this issue and wanted an opportunity to understand how this subcommand works and see if I can add this feature :^)

I tested both with --skip-db and without and it appears to not add any database-specific files to the generated archive i.e. I don't see a gitea-db.sql or gitea.db file:

$ TAGS="bindata sqlite sqlite_unlock_notify" make backend
Running go generate...
bindata for migration already up-to-date
bindata for options already up-to-date
bindata for public already up-to-date
bindata for templates already up-to-date
$ ./gitea dump --skip-db
2024/04/20 01:16:11 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/04/20 01:16:11 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/attachments
2024/04/20 01:16:11 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/avatars
2024/04/20 01:16:11 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-avatars
2024/04/20 01:16:11 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/lfs
2024/04/20 01:16:11 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-archive
2024/04/20 01:16:11 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/packages
2024/04/20 01:16:11 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_log
2024/04/20 01:16:11 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_artifacts
2024/04/20 01:16:11 cmd/dump.go:172:runDump() [I] Dumping local repositories... /workspaces/gitea/data/gitea-repositories
2024/04/20 01:16:11 cmd/dump.go:195:runDump() [I] Skipping database
2024/04/20 01:16:11 cmd/dump.go:229:runDump() [I] Adding custom configuration file from /workspaces/gitea/custom/conf/app.ini
2024/04/20 01:16:11 cmd/dump.go:256:runDump() [I] Packing data directory.../workspaces/gitea/data
2024/04/20 01:16:11 cmd/dump.go:335:runDump() [I] Finish dumping in file /workspaces/gitea/gitea-dump-1713575771.zip
$ unzip /workspaces/gitea/gitea-dump-1713575771.zip -d example
Archive:  /workspaces/gitea/gitea-dump-1713575771.zip
. . .
$ ls example/
app.ini  custom  data  repos
$ ls example/data/
actions_artifacts  actions_log  avatars  home  indexers  jwt  queues  repo-archive  repo-avatars  tmp

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Apr 20, 2024
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 20, 2024
@github-actions github-actions bot added modifies/go Pull requests that update Go code modifies/cli PR changes something on the CLI, i.e. gitea doctor or gitea admin labels Apr 20, 2024
}
if ctx.Bool("skip-db") {
// Ensure that we don't dump the database file that may reside in setting.AppDataPath or elsewhere.
dumper.GlobalExcludeAbsPath(setting.Database.Path)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it safe to assume that setting.Database.Path is an absolute path?

Copy link
Contributor Author

@kemzeb kemzeb Apr 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I seen other areas that perform a ctx.IsSet() before a ctx.Bool(), but after reading the docs associated to the Bool() method it should return false if --skip-db is not provided; so I decided to only use this function. Please let me know if this is a problem

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the path maybe empty? Whether it will affect the result?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From looking at GlobalExcludeAbsPath() (see here), if it was passed the empty string, it would essentially append "." to dumper.globalExcludeAbsPaths.

Even though it's not an absolute path, it doesn't look like it would affect the result as we if we add a "." to this slice any subsequent paths that are just the empty string would also be cleaned to be "." and therefore be excluded since that string is in the slice. If you feel differently, please let me know and I'll add an empty string check for this

if _, err := os.Stat(tmpDir); os.IsNotExist(err) {
fatal("Path does not exist: %s", tmpDir)
}
if ctx.Bool("skip-db") {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't add a comment to the specific code snippet for some reason, but should we avoid firing up the xorm engine if we are skipping the database (so lines 140-146 as of the current snapshot of my branch). After looking around I think we can add it into my else branch but I'm unsure if it is needed anywhere else in the function.

@kemzeb
Copy link
Contributor Author

kemzeb commented Apr 20, 2024

An idea unrelated to this PR I had while working on this. A --skip option could be added where uers can include what they want to skip as a StringSliceFlag i.e. --skip db,repository,custom-dir. Though we would need to document them all in a single option; not sure if that may be a problem.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Apr 20, 2024
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 21, 2024
@lunny lunny added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Apr 21, 2024
@silverwind silverwind merged commit 0606284 into go-gitea:main Apr 21, 2024
26 checks passed
@GiteaBot GiteaBot added this to the 1.23.0 milestone Apr 21, 2024
@GiteaBot GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Apr 21, 2024
@silverwind silverwind added the backport/v1.22 This PR should be backported to Gitea 1.22 label Apr 21, 2024
@kemzeb kemzeb deleted the cmd-dump-add-skip-db-option branch April 21, 2024 21:38
GiteaBot added a commit to GiteaBot/gitea that referenced this pull request Apr 21, 2024
Attempts to resolve go-gitea#28720.

---

Note that I am not a Gitea administrator so I don't normally use the
gitea CLI. Just saw this issue and wanted an opportunity to understand
how this subcommand works and see if I can add this feature :^)

I tested both with `--skip-db` and without and it appears to not add any
database-specific files to the generated archive i.e. I don't see a
`gitea-db.sql` or `gitea.db` file:
```console
$ TAGS="bindata sqlite sqlite_unlock_notify" make backend
Running go generate...
bindata for migration already up-to-date
bindata for options already up-to-date
bindata for public already up-to-date
bindata for templates already up-to-date
$ ./gitea dump --skip-db
2024/04/20 01:16:11 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/04/20 01:16:11 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/attachments
2024/04/20 01:16:11 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/avatars
2024/04/20 01:16:11 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-avatars
2024/04/20 01:16:11 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/lfs
2024/04/20 01:16:11 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-archive
2024/04/20 01:16:11 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/packages
2024/04/20 01:16:11 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_log
2024/04/20 01:16:11 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_artifacts
2024/04/20 01:16:11 cmd/dump.go:172:runDump() [I] Dumping local repositories... /workspaces/gitea/data/gitea-repositories
2024/04/20 01:16:11 cmd/dump.go:195:runDump() [I] Skipping database
2024/04/20 01:16:11 cmd/dump.go:229:runDump() [I] Adding custom configuration file from /workspaces/gitea/custom/conf/app.ini
2024/04/20 01:16:11 cmd/dump.go:256:runDump() [I] Packing data directory.../workspaces/gitea/data
2024/04/20 01:16:11 cmd/dump.go:335:runDump() [I] Finish dumping in file /workspaces/gitea/gitea-dump-1713575771.zip
$ unzip /workspaces/gitea/gitea-dump-1713575771.zip -d example
Archive:  /workspaces/gitea/gitea-dump-1713575771.zip
. . .
$ ls example/
app.ini  custom  data  repos
$ ls example/data/
actions_artifacts  actions_log  avatars  home  indexers  jwt  queues  repo-archive  repo-avatars  tmp
```

Co-authored-by: Giteabot <teabot@gitea.io>
@GiteaBot GiteaBot added the backport/done All backports for this PR have been created label Apr 21, 2024
lunny pushed a commit that referenced this pull request Apr 22, 2024
Backport #30613 by @kemzeb

Attempts to resolve #28720.

Co-authored-by: Kemal Zebari <60799661+kemzeb@users.noreply.github.com>
zjjhot added a commit to zjjhot/gitea that referenced this pull request Apr 22, 2024
* giteaofficial/main:
  [skip ci] Updated licenses and gitignores
  Use correct hash for "git update-index" (go-gitea#30626)
  Fix repo home UI when there is no repo description (go-gitea#30552)
  Fix dropdown text ellipsis (go-gitea#30628)
  fix(api): refactor branch and tag existence checks (go-gitea#30618)
  Add --skip-db option to dump command (go-gitea#30613)
  Fix flash on dashboard (go-gitea#30572)
  chore: use errors.New to replace fmt.Errorf with no parameters will much better (go-gitea#30621)
  Fix issue comment form and quick-submit (go-gitea#30623)
  Use maintained gziphandler (go-gitea#30592)
  [skip ci] Updated translations via Crowdin
  Fix package list performance (go-gitea#30520)
  Clarify permission "HasAccess" behavior (go-gitea#30585)
  Fix links in PyPI Simple Repository API page (go-gitea#30594)
  Use action user as the trigger user of schedules (go-gitea#30581)
  Fix commit file status parser (go-gitea#30602)
  Fix HEAD method for robots.txt (go-gitea#30603)
silverwind added a commit to silverwind/gitea that referenced this pull request Apr 23, 2024
* origin/main:
  Interpolate runs-on with variables when scheduling tasks (go-gitea#30640)
  Initial support for colorblindness-friendly themes (go-gitea#30625)
  Fix flash message for flex-container (go-gitea#30657)
  Perform Newest sort type correctly when sorting issues (go-gitea#30644)
  Fix project name wrapping, remove horizontal margin on header (go-gitea#30631)
  Add a db consistency check to remove runners that do not belong to a repository (go-gitea#30614)
  Fix wrong table name (go-gitea#30557)
  Fix compare api swagger (go-gitea#30648)
  [skip ci] Updated translations via Crowdin
  Fix queue test (go-gitea#30646)
  Enable jquery-related eslint rules that have no violations (go-gitea#30632)
  Enable more `revive` linter rules (go-gitea#30608)
  Remove obsolete CSS text classes (go-gitea#30576)
  Hide diff stats on empty PRs (go-gitea#30629)
  [skip ci] Updated licenses and gitignores
  Use correct hash for "git update-index" (go-gitea#30626)
  Fix repo home UI when there is no repo description (go-gitea#30552)
  Fix dropdown text ellipsis (go-gitea#30628)
  fix(api): refactor branch and tag existence checks (go-gitea#30618)
  Add --skip-db option to dump command (go-gitea#30613)
@wxiaoguang wxiaoguang added the skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. label Apr 27, 2024
@go-gitea go-gitea locked as resolved and limited conversation to collaborators Jul 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/done All backports for this PR have been created backport/v1.22 This PR should be backported to Gitea 1.22 lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/cli PR changes something on the CLI, i.e. gitea doctor or gitea admin modifies/go Pull requests that update Go code size/M Denotes a PR that changes 30-99 lines, ignoring generated files. skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a --skip-db option to gitea dump
5 participants