Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bucket storage support #9567

Open
wants to merge 23 commits into
base: master
from
Open

Add bucket storage support #9567

wants to merge 23 commits into from

Conversation

@cuonglm
Copy link

cuonglm commented Dec 31, 2019

This is a follow up PR for #7795, cherry-pick those commits and add:

  • Abstract filesystem layer for avatars/attachments/lfs objects, using afero
  • Fix some issue with test, error checking
  • Add tests for bucket storage using minio (for s3 compatibility) and fake-gcs-server (for GCS)
  • Add storage-migrate command to migrate from local to cloud storage and vice versa.

Running:

gitea storage-migrate --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

will migrate from local to cloud storage, and:

gitea storage-migrate --local --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

will migrate back from cloud to local.

@cuonglm cuonglm mentioned this pull request Dec 31, 2019
2 of 2 tasks complete
@cuonglm cuonglm force-pushed the patcito:master branch 6 times, most recently from c13ed6b to 0b24589 Dec 31, 2019
@lunny lunny added the kind/feature label Jan 1, 2020
@lunny lunny added this to the 1.12.0 milestone Jan 1, 2020
- refs/heads/master
- "refs/tags/**"
- "refs/pull/**"
- refs/heads/master

This comment has been minimized.

Copy link
@lunny

lunny Jan 1, 2020

Member

Why change this?

This comment has been minimized.

Copy link
@cuonglm

cuonglm Jan 1, 2020

Author

@lunny It’s auto format by the editor. Both are legal yaml syntax. AFAICT, there’re some yaml parsers out there which rejects the former, but not the later.

This comment has been minimized.

Copy link
@lunny

lunny Jan 2, 2020

Member

So your formatter is different from me. :(

This comment has been minimized.

Copy link
@cuonglm

cuonglm Jan 2, 2020

Author

@lunny Actually, I did nothing with the config, as it's the default in Goland 😄If it's not ok to change, I will revet.

@GiteaBot GiteaBot added the lgtm/need 2 label Jan 1, 2020
Copy link
Member

lunny left a comment

I think all temp dir should be local file system.

cmd/dump.go Outdated Show resolved Hide resolved
contrib/pr/checkout.go Outdated Show resolved Hide resolved
Copy link
Member

techknowlogick left a comment

Thank you for this PR 😄 very excited for it.

I've gone through and noticed a few items with import structures, and noted a few of them (there are others that I haven't noted, but should still also be resolved)

@cuonglm cuonglm force-pushed the patcito:master branch from 0b24589 to 68c975e Jan 1, 2020
@cuonglm cuonglm requested review from lunny and techknowlogick Jan 1, 2020
models/attachment.go Outdated Show resolved Hide resolved
@cuonglm cuonglm force-pushed the patcito:master branch from 68c975e to 8140fbd Jan 1, 2020
@cuonglm cuonglm requested a review from lunny Jan 1, 2020
cmd/storage.go Outdated Show resolved Hide resolved
@cuonglm cuonglm force-pushed the patcito:master branch from 8140fbd to f28f25e Jan 1, 2020
@cuonglm cuonglm requested a review from lunny Jan 1, 2020
@cuonglm cuonglm force-pushed the patcito:master branch from ece7231 to c867c97 Jan 15, 2020
@cuonglm cuonglm requested a review from lunny Jan 15, 2020
@cuonglm cuonglm force-pushed the patcito:master branch 2 times, most recently from 6ab1dd5 to df4a49a Jan 15, 2020
@lunny

This comment has been minimized.

Copy link
Member

lunny commented Jan 15, 2020

The two files options/gitignore/JBoss and another one should be reverted.

masudur-rahman and others added 23 commits Sep 3, 2019
Signed-off-by: Masudur Rahman <masud@appscode.com>
The Write is only guaranteed to have succeeded if Close returns no error.
This is necessary for testing client without authentication.
 - Use the right public-host for fake-gcs-server
 - Running TestGit in separated command to prevent using all available
 ports of the container.
ContentStore.Verify always return non-nil error, which seems to be wrong
behavior. In case error occured, we should return instead of masking it
like current behavior.
GCS client calls are asynchronous, with large upload files, it causes
the CI container run out of resources quickly.
We should not exit gitea if we can not create local file, just return
the error to the caller instead.
This PR adds subcommand "storage-migrate" command to allow migration
from local to cloud storage and vice versa.

To migrate from local to cloud:

	gitea storage-migrate --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

To migrate from cloud to local:

	gitea storage-migrate --local --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

Absolute paths are not supported, since when storage.OpenBucket always
assumes local path if the path is absolute.
To make sure response body is closed immediately.
@cuonglm

This comment has been minimized.

Copy link
Author

cuonglm commented Jan 15, 2020

The two files options/gitignore/JBoss and another one should be reverted.

Done

@cuonglm cuonglm force-pushed the patcito:master branch from df4a49a to 636a1a7 Jan 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.