Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure blob storage support #30995

Merged
merged 56 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
0a0c188
first
yp05327 Jun 23, 2023
fce996a
fix lint
yp05327 Jun 23, 2023
efc0ed7
add ci test
yp05327 Jun 23, 2023
f3d994d
Merge branch 'main' into azure-blob-storage-support
yp05327 Jun 23, 2023
888cfa5
fix url
yp05327 Jun 23, 2023
ab798ec
remove
yp05327 Jul 6, 2023
48d98e7
add todo
yp05327 Jul 6, 2023
3901076
Merge branch 'main' into azure-blob-storage-support
yp05327 Jul 25, 2023
ec57c06
Merge branch 'main' into azure-blob-storage-support
yp05327 Nov 17, 2023
2c00368
update azure sdk
yp05327 Nov 17, 2023
f45367f
update
yp05327 Nov 17, 2023
e03829f
rename to azureblob
yp05327 Nov 17, 2023
5ade526
support direct url for azure blob
yp05327 Nov 17, 2023
3b49d30
fix
yp05327 Nov 17, 2023
bc27561
improve
yp05327 Nov 20, 2023
e438513
imporve
yp05327 Nov 20, 2023
8e007dc
improve azure blob url
yp05327 Nov 24, 2023
5faecb9
add path test
yp05327 Nov 24, 2023
f7e42d2
fix lint
yp05327 Nov 24, 2023
2a8add1
Merge branch 'azure-blob-storage-support' of github.com:yp05327/gitea…
lunny May 2, 2024
14a194c
Add some tests for configuration of azure blob storage support
lunny May 2, 2024
c6e3538
upgrade azure blob version
lunny May 6, 2024
594cbeb
Merge branch 'main' into lunny/azure_blob_storage
lunny May 11, 2024
95ab4d1
Fix download blob
lunny May 13, 2024
de67cea
Fix bug
lunny May 16, 2024
4fc10e8
Merge branch 'main' into lunny/azure_blob_storage
lunny May 16, 2024
755f13a
Add documentations
lunny May 16, 2024
37c2ffc
Add azureblob test for mssql tests
lunny May 16, 2024
1d0d1ed
Fix ci
lunny May 16, 2024
8dcf589
Fix ci
lunny May 16, 2024
c3f1090
Fix read size
lunny May 17, 2024
9741953
Merge branch 'main' into lunny/azure_blob_storage
lunny May 17, 2024
22a247a
Fix test bug
lunny May 17, 2024
929a4e3
Fix bug
lunny May 17, 2024
0f32f6b
Fix servedirect
lunny May 17, 2024
8a01693
Remove comment code
lunny May 17, 2024
068bb24
Fix test
lunny May 17, 2024
15c439d
Fix removing non-exist attachment
lunny May 18, 2024
50bf919
improve test
lunny May 19, 2024
ca6a1dc
package version should not have space
lunny May 20, 2024
c23e531
Fix the problem that azurite's special check for SASURL
lunny May 21, 2024
4b88ab4
Fix test
lunny May 21, 2024
baca070
Merge branch 'main' into lunny/azure_blob_storage
lunny May 21, 2024
4511c13
Revert unnecessary change
lunny May 22, 2024
3646822
Merge branch 'main' into lunny/azure_blob_storage
lunny May 22, 2024
a3c56dd
Apply suggestions from code review
lunny May 23, 2024
e76bf65
Update documentations
lunny May 23, 2024
391fd1e
Fix bug
lunny May 27, 2024
8fb3532
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 29, 2024
6c35486
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 29, 2024
a28b05e
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 29, 2024
066f05f
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 29, 2024
f2ae422
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 29, 2024
55de09c
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 30, 2024
1ef76dc
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 30, 2024
f81c683
Merge branch 'main' into lunny/azure_blob_storage
GiteaBot May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@
"ms-azuretools.vscode-docker",
"vitest.explorer",
"qwtel.sqlite-viewer",
"GitHub.vscode-pull-request-github"
"GitHub.vscode-pull-request-github",
"Azurite.azurite"
]
}
},
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/pull-db-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,14 +119,18 @@ jobs:
MINIO_SECRET_KEY: 12345678
ports:
- "9000:9000"
devstoreaccount1.azurite.local: # https://github.com/Azure/Azurite/issues/1583
image: mcr.microsoft.com/azure-storage/azurite:latest
ports:
- 10000:10000
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
check-latest: true
- name: Add hosts to /etc/hosts
run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mysql elasticsearch meilisearch smtpimap" | sudo tee -a /etc/hosts'
run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 minio devstoreaccount1.azurite.local mysql elasticsearch meilisearch smtpimap" | sudo tee -a /etc/hosts'
- run: make deps-backend
- run: make backend
env:
Expand Down Expand Up @@ -204,14 +208,18 @@ jobs:
SA_PASSWORD: MwantsaSecurePassword1
ports:
- "1433:1433"
devstoreaccount1.azurite.local: # https://github.com/Azure/Azurite/issues/1583
image: mcr.microsoft.com/azure-storage/azurite:latest
ports:
- 10000:10000
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
check-latest: true
- name: Add hosts to /etc/hosts
run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mssql" | sudo tee -a /etc/hosts'
run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mssql devstoreaccount1.azurite.local" | sudo tee -a /etc/hosts'
- run: make deps-backend
- run: make backend
env:
Expand Down
15 changes: 15 additions & 0 deletions assets/go-licenses.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

41 changes: 40 additions & 1 deletion cmd/migrate_storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,15 @@ var CmdMigrateStorage = &cli.Command{
Name: "storage",
Aliases: []string{"s"},
Value: "",
Usage: "New storage type: local (default) or minio",
Usage: "New storage type: local (default), minio or azureblob",
},
&cli.StringFlag{
Name: "path",
Aliases: []string{"p"},
Value: "",
Usage: "New storage placement if store is local (leave blank for default)",
},
// Minio Storage special configurations
&cli.StringFlag{
Name: "minio-endpoint",
Value: "",
Expand Down Expand Up @@ -96,6 +97,32 @@ var CmdMigrateStorage = &cli.Command{
Value: "",
Usage: "Minio bucket lookup type",
},
// Azure Blob Storage special configurations
&cli.StringFlag{
Name: "azureblob-endpoint",
Value: "",
Usage: "Azure Blob storage endpoint",
},
&cli.StringFlag{
Name: "azureblob-account-name",
Value: "",
Usage: "Azure Blob storage account name",
},
&cli.StringFlag{
Name: "azureblob-account-key",
Value: "",
Usage: "Azure Blob storage account key",
},
&cli.StringFlag{
Name: "azureblob-container",
Value: "",
Usage: "Azure Blob storage container",
},
&cli.StringFlag{
Name: "azureblob-base-path",
Value: "",
Usage: "Azure Blob storage base path",
},
},
}

Expand Down Expand Up @@ -228,6 +255,18 @@ func runMigrateStorage(ctx *cli.Context) error {
BucketLookUpType: ctx.String("minio-bucket-lookup-type"),
},
})
case string(setting.AzureBlobStorageType):
dstStorage, err = storage.NewAzureBlobStorage(
stdCtx,
&setting.Storage{
AzureBlobConfig: setting.AzureBlobStorageConfig{
Endpoint: ctx.String("azureblob-endpoint"),
AccountName: ctx.String("azureblob-account-name"),
AccountKey: ctx.String("azureblob-account-key"),
Container: ctx.String("azureblob-container"),
BasePath: ctx.String("azureblob-base-path"),
},
})
default:
return fmt.Errorf("unsupported storage type: %s", ctx.String("storage"))
}
Expand Down
Loading