Clear the data/log for jobs in Gitea Actions that are too old #24256
Comments
wolfogre
added
type/proposal
|
Maybe a related question? |
|
I was now looking for this 😊 |
|
Maybe we can compress as zip, so that when display via HTTP, it can be unzip by web browser. |
|
Maybe, but I think most important is to be able to purge too, with some Retention setting (e.g. older than 1 month, keep last 3, etc), so that the storage doesn't grow forever. |
|
+1 for purge ability; this is my use case: I'm managing an internal Gitea server, and I'm going to make Actions available to my co-worker devs. Since there's no other way to test push Actions than... pushing, I've played with them by "trial and error" 😉. To make it worst, I suppose that once Actions will be made available to anyone on my server, my co-workers will also start testing them the same way... with the same result. And finally, once in production, there's no need for us to keep any of repos Actions logs longer then 2wks. |
|
Related #26219 (Delete workflow runs) (manually) |
|
about the Artifacts would it be possible to add a little rubbish icon next to the file name so they can be deleted manually? (talking about artifacts created by actions and uploaded via actions/upload-artifact) |
|
For me, 7 days is more than enough |
Would be much better to have this interval configurable per repository. |
|
After deleting the old action logs, the spinners don't stop and there are no messages to indicate the log no longer exists. |
Append to me as well when the runner dies because it could not download an action. |
@bilogic How do you clear the old action logs? Just remove them from disk/storage? If so, I would say it's safe to do that to free up space. However, the old jobs cannot be displayed properly on the UI since there's no code for the inconsistency. |
|
@wolfogre Yes, I just removed from storage, but that was because I had a backup. And yes, my image was to illustrate the need to report the situation if the logs are deleted. |
Part of #24256. Clear up old action logs to free up storage space. Users will see a message indicating that the log has been cleared if they view old tasks. <img width="1361" alt="image" src="https://github.com/user-attachments/assets/9f0f3a3a-bc5a-402f-90ca-49282d196c22"> Docs: https://gitea.com/gitea/docs/pulls/40 --------- Co-authored-by: silverwind <me@silverwind.io>
Support compression for Actions logs to save storage space and bandwidth. Inspired by #24256 (comment) The biggest challenge is that the compression format should support [seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md). So when users are viewing a part of the log lines, Gitea doesn't need to download the whole compressed file and decompress it. That means gzip cannot help here. And I did research, there aren't too many choices, like bgzip and xz, but I think zstd is the most popular one. It has an implementation in Golang with [zstd](https://github.com/klauspost/compress/tree/master/zstd) and [zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go), and what is better is that it has good compatibility: a seekable format zstd file can be read by a regular zstd reader. This PR introduces a new package `zstd` to combine and wrap the two packages, to provide a unified and easy-to-use API. And a new setting `LOG_COMPRESSION` is added to the config, although I don't see any reason why not to use compression, I think's it's a good idea to keep the default with `none` to be consistent with old versions. `LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as an extension to the file name, so Gitea can determine if it needs decompression according to the file name when reading. Old files will keep the format since it's not worth converting them, as they will be cleared after #31735. <img width="541" alt="image" src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
Support compression for Actions logs to save storage space and bandwidth. Inspired by go-gitea/gitea#24256 (comment) The biggest challenge is that the compression format should support [seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md). So when users are viewing a part of the log lines, Gitea doesn't need to download the whole compressed file and decompress it. That means gzip cannot help here. And I did research, there aren't too many choices, like bgzip and xz, but I think zstd is the most popular one. It has an implementation in Golang with [zstd](https://github.com/klauspost/compress/tree/master/zstd) and [zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go), and what is better is that it has good compatibility: a seekable format zstd file can be read by a regular zstd reader. This PR introduces a new package `zstd` to combine and wrap the two packages, to provide a unified and easy-to-use API. And a new setting `LOG_COMPRESSION` is added to the config, although I don't see any reason why not to use compression, I think's it's a good idea to keep the default with `none` to be consistent with old versions. `LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as an extension to the file name, so Gitea can determine if it needs decompression according to the file name when reading. Old files will keep the format since it's not worth converting them, as they will be cleared after #31735. <img width="541" alt="image" src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9"> (cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588) Conflicts: assets/go-licenses.json go.mod go.sum resolved with make tidy
Feature Description
Jobs that have been done for too long no longer need to retain data/log, so it is necessary to regularly clean them up.
Screenshots
No response
The text was updated successfully, but these errors were encountered: