Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix merging artifact chunks error when minio storage basepath is set #28555

Merged
merged 4 commits into from Dec 21, 2023
Merged

Fix merging artifact chunks error when minio storage basepath is set #28555

merged 4 commits into from Dec 21, 2023

Conversation

fuxiaohei
Copy link
Contributor

@fuxiaohei fuxiaohei commented Dec 20, 2023 •
edited

Related to #28279

When merging artifact chunks, it lists chunks from storage. When storage is minio, chunk's path contains MINIO_BASE_PATH that makes merging break.

So trim the MINIO_BASE_PATH when handle chunks.

Update the chunk file's basename to retain necessary information. It ensures that the directory in the chunk's path remains unaffected.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Dec 20, 2023
@pull-request-size pull-request-size bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Dec 20, 2023
@github-actions github-actions bot added the modifies/api This PR adds API routes or modifies them label Dec 20, 2023
@6543 6543 added type/bug topic/gitea-actions related to the actions of Gitea labels Dec 20, 2023
delvh
delvh approved these changes Dec 20, 2023
View reviewed changes
Copy link
Member

@delvh delvh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took me a bit to understand the fix, but I understand it now.

However, what really confuses me: Aren't PR description and fix completely different?

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Dec 20, 2023
@fuxiaohei fuxiaohei changed the title fix: trim minio basepath when merging chunks Fix merging artifact chunks error when minio storage basepath is set Dec 20, 2023
wxiaoguang
wxiaoguang reviewed Dec 20, 2023
View reviewed changes
routers/api/actions/artifacts_chunks.go Outdated
baseName := filepath.Base(fpath)
// when read chunks from storage, it only contains storage dir and basename,
// no matter the subdirectory setting in storage config
item := chunkFileItem{Path: path.Join(storageDir, baseName)}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It confused me that it sometimes calls filepath (line 74) while sometimes calls path (here). It looks quite fragile to me.

At least, IMO it needs to normalize the "paths" before using it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second path.Join is not necessary. It saves chunk as filename tmp/xxx.chunk in above. So just storageDir +"/" + basename is enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have difficulty to understand whether it is right to do these magic path operations.

Could it clarify these "path kinds"? If a path is a filesystem path, then it needs filepath package, if it is a web path (or a web path converted from filepath), then it needs util.PathJoinRelX.

Copy link
Contributor Author

@fuxiaohei fuxiaohei Dec 21, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

baseName := filepath.Base(fpath) is the real fix. fpath is from Storage and can be filesystem path so use filepath to get basename.

chunkFileItem.Path is a key to read and write file for Storage. When saving chunk, it uses fmt.Sprintf("tmp%d/%d-%d-%d-%d.chunk", runID, runID, artifact.ID, start, end) with / slash when saving into Storage.

Using storageDir +"/" + basename is enough in item.Path to read from Storage. Storage module can handle os difference.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Dec 21, 2023
@lunny lunny added reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. backport/v1.21 This PR should be backported to Gitea 1.21 labels Dec 21, 2023
@lunny lunny enabled auto-merge (squash) December 21, 2023 06:30
@lunny lunny merged commit fe5a616 into go-gitea:main Dec 21, 2023
25 checks passed
@GiteaBot GiteaBot added this to the 1.22.0 milestone Dec 21, 2023
@GiteaBot GiteaBot mentioned this pull request Dec 21, 2023
Merged
GiteaBot pushed a commit to GiteaBot/gitea that referenced this pull request Dec 21, 2023
@fuxiaohei @GiteaBot
Fix merging artifact chunks error when minio storage basepath is set (g…
ac7ad10 
…o-gitea#28555)

Related to  go-gitea#28279

When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.

<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>

Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.
@GiteaBot GiteaBot added backport/done All backports for this PR have been created and removed reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. labels Dec 21, 2023
lunny pushed a commit that referenced this pull request Dec 21, 2023
@GiteaBot @fuxiaohei
Fix merging artifact chunks error when minio storage basepath is set (#…
8ca32dc 
…28555) (#28568)

Backport #28555 by @fuxiaohei

Related to  #28279

When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.

<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>

Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.

Co-authored-by: FuXiaoHei <fuxiaohei@vip.qq.com>
zjjhot added a commit to zjjhot/gitea that referenced this pull request Dec 22, 2023
@zjjhot
Merge remote-tracking branch 'giteaofficial/main'
9296ce0 
* giteaofficial/main:
  Add more ways to try (go-gitea#28581)
  Convert to url auth to header auth in tests (go-gitea#28484)
  Fix 500 error of searching commits (go-gitea#28576)
  improve possible performance bottleneck (go-gitea#28547)
  Use information from previous blame parts (go-gitea#28572)
  Make offline mode as default to no connect external avatar service by default (go-gitea#28548)
  Fix merging artifact chunks error when minio storage basepath is set (go-gitea#28555)
  feat: bump `dessant/lock-threads` and `actions/setup-go` to use nodejs20 runtime (go-gitea#28565)
  Update actions document about comparsion as Github Actions (go-gitea#28560)
  Fix inperformant query on retrifing review from database. (go-gitea#28552)
  Fix the issue ref rendering for wiki (go-gitea#28556)
  Add missing head of lfs client batch (go-gitea#28550)
  [skip ci] Updated translations via Crowdin
  Remove deadcode under models/issues (go-gitea#28536)
  Always enable caches (go-gitea#28527)
  Improve ObjectFormat interface (go-gitea#28496)
fuxiaohei added a commit to fuxiaohei/gitea that referenced this pull request Jan 17, 2024
@fuxiaohei
Fix merging artifact chunks error when minio storage basepath is set (g…
b452dc3 
…o-gitea#28555)

Related to  go-gitea#28279

When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.

<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>

Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.
silverwind pushed a commit to silverwind/gitea that referenced this pull request Feb 20, 2024
@fuxiaohei @silverwind
Fix merging artifact chunks error when minio storage basepath is set (g…
128224b 
…o-gitea#28555)

Related to  go-gitea#28279

When merging artifact chunks, it lists chunks from storage. When storage
is minio, chunk's path contains `MINIO_BASE_PATH` that makes merging
break.

<del>So trim the `MINIO_BASE_PATH` when handle chunks.</del>

Update the chunk file's basename to retain necessary information. It
ensures that the directory in the chunk's path remains unaffected.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@wxiaoguang wxiaoguang wxiaoguang left review comments

@lunny lunny lunny approved these changes

@delvh delvh delvh approved these changes

Assignees
No one assigned
Labels
backport/done All backports for this PR have been created backport/v1.21 This PR should be backported to Gitea 1.21 lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/api This PR adds API routes or modifies them size/S Denotes a PR that changes 10-29 lines, ignoring generated files. topic/gitea-actions related to the actions of Gitea type/bug
Projects
None yet
Milestone
1.22.0
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@fuxiaohei @lunny @wxiaoguang @delvh @GiteaBot @6543