Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s2: Add AsyncFlush method: Complete the block without flushing #927

Merged
merged 2 commits into from Feb 12, 2024
Merged

s2: Add AsyncFlush method: Complete the block without flushing #927

merged 2 commits into from Feb 12, 2024

Conversation

Jille
Copy link
Contributor

@Jille Jille commented Feb 9, 2024

My use case is to transfer a large compressed S2 stream with a few changes very often. To get a small diff I want to end blocks at application decided points rather than at byte offsets. This allows me to remove the first byte without every single block changing.

Flush() works for this, but it limits concurrency because it waits for the last block to be compressed rather than allowing that asynchronously.

So I'd like to propose AsyncFlush, which flushes the buffer to a block, but doesn't flush the block to the io.Writer.

There were actually a few places in the s2 code that also wanted to end the block, but didn't necessary want to flush to the writer.

@Jille
s2: Add AsyncFlush method: Complete the block without flushing
2c36775 
My use case is to transfer a large compressed S2 stream with a few
changes very often. To get a small diff I want to end blocks at
application decided points rather than at byte offsets. This allows me
to remove the first byte without every single block changing.

Flush() works for this, but it limits concurrency because it waits for
the last block to be compressed rather than allowing that
asynchronously.

So I'd like to propose AsyncFlush, which flushes the buffer to a block,
but doesn't flush the block to the io.Writer.

There were actually a few places in the s2 code that also wanted to end
the block, but didn't necessary want to flush to the writer.
klauspost
klauspost reviewed Feb 9, 2024
View reviewed changes
Copy link
Owner

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonderful change. The only thing I would tweak slightly is the documentation.

s2/writer.go Outdated Show resolved Hide resolved
@Jille @klauspost
Update s2/writer.go
09a048f 
Co-authored-by: Klaus Post <klauspost@gmail.com>
@Jille
Copy link
Contributor Author

Jille commented Feb 12, 2024

Sounds good to me! Thanks :)

klauspost
klauspost approved these changes Feb 12, 2024
View reviewed changes
Copy link
Owner

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@klauspost klauspost merged commit 5895eb4 into klauspost:master Feb 12, 2024
18 checks passed
codeboten pushed a commit to open-telemetry/opentelemetry-collector that referenced this pull request Feb 28, 2024
@renovate @opentelemetrybot @dmitryax
Update module github.com/klauspost/compress to v1.17.7 (#9644)
4b75e34 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/klauspost/compress](https://togithub.com/klauspost/compress)
| `v1.17.6` -> `v1.17.7` |
[![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

###
[`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare
Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

#### What's Changed

- s2: Add AsyncFlush method: Complete the block without flushing by
[@&#8203;Jille](https://togithub.com/Jille) in
[klauspost/compress#927
- s2: Fix literal+repeat exceeds dst crash by
[@&#8203;klauspost](https://togithub.com/klauspost) in
[klauspost/compress#930

**Full Changelog**:
klauspost/compress@v1.17.6...v1.17.7

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMTIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIxMi4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
kodiakhq bot pushed a commit to cloudquery/codegen that referenced this pull request Mar 1, 2024
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.17.7 (#159)
118a3b5 
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | patch | `v1.17.5` -> `v1.17.7` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

#### What's Changed

-   s2: Add AsyncFlush method: Complete the block without flushing by [@&#8203;Jille](https://togithub.com/Jille) in [klauspost/compress#927
-   s2: Fix literal+repeat exceeds dst crash by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#930

**Full Changelog**: klauspost/compress@v1.17.6...v1.17.7

### [`v1.17.6`](https://togithub.com/klauspost/compress/releases/tag/v1.17.6)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.5...v1.17.6)

#### What's Changed

-   zstd: Fix incorrect repeat coding in best mode by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#923
-   s2: Fix DecodeConcurrent deadlock on errors by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#925
-   build: Remove garble compiler by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#924

**Full Changelog**: klauspost/compress@v1.17.5...v1.17.6

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMjEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
kodiakhq bot pushed a commit to cloudquery/filetypes that referenced this pull request Mar 1, 2024
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.17.7 (#449)
7cfe20f 
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | patch | `v1.17.5` -> `v1.17.7` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

#### What's Changed

-   s2: Add AsyncFlush method: Complete the block without flushing by [@&#8203;Jille](https://togithub.com/Jille) in [klauspost/compress#927
-   s2: Fix literal+repeat exceeds dst crash by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#930

**Full Changelog**: klauspost/compress@v1.17.6...v1.17.7

### [`v1.17.6`](https://togithub.com/klauspost/compress/releases/tag/v1.17.6)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.5...v1.17.6)

#### What's Changed

-   zstd: Fix incorrect repeat coding in best mode by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#923
-   s2: Fix DecodeConcurrent deadlock on errors by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#925
-   build: Remove garble compiler by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#924

**Full Changelog**: klauspost/compress@v1.17.5...v1.17.6

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMjEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
kodiakhq bot pushed a commit to cloudquery/plugin-pb-go that referenced this pull request Mar 1, 2024
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.17.7 (#265)
b985ac4 
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | patch | `v1.17.5` -> `v1.17.7` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

#### What's Changed

-   s2: Add AsyncFlush method: Complete the block without flushing by [@&#8203;Jille](https://togithub.com/Jille) in [klauspost/compress#927
-   s2: Fix literal+repeat exceeds dst crash by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#930

**Full Changelog**: klauspost/compress@v1.17.6...v1.17.7

### [`v1.17.6`](https://togithub.com/klauspost/compress/releases/tag/v1.17.6)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.5...v1.17.6)

#### What's Changed

-   zstd: Fix incorrect repeat coding in best mode by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#923
-   s2: Fix DecodeConcurrent deadlock on errors by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#925
-   build: Remove garble compiler by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#924

**Full Changelog**: klauspost/compress@v1.17.5...v1.17.6

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMjEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
kodiakhq bot pushed a commit to cloudquery/plugin-sdk that referenced this pull request Mar 1, 2024
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.17.7 (#1544
4e04027 
)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | patch | `v1.17.5` -> `v1.17.7` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

#### What's Changed

-   s2: Add AsyncFlush method: Complete the block without flushing by [@&#8203;Jille](https://togithub.com/Jille) in [klauspost/compress#927
-   s2: Fix literal+repeat exceeds dst crash by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#930

**Full Changelog**: klauspost/compress@v1.17.6...v1.17.7

### [`v1.17.6`](https://togithub.com/klauspost/compress/releases/tag/v1.17.6)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.5...v1.17.6)

#### What's Changed

-   zstd: Fix incorrect repeat coding in best mode by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#923
-   s2: Fix DecodeConcurrent deadlock on errors by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#925
-   build: Remove garble compiler by [@&#8203;klauspost](https://togithub.com/klauspost) in [klauspost/compress#924

**Full Changelog**: klauspost/compress@v1.17.5...v1.17.6

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMjEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
codeboten pushed a commit to open-telemetry/opentelemetry-collector-contrib that referenced this pull request Mar 1, 2024
@renovate @opentelemetrybot
fix(deps): update module github.com/klauspost/compress to v1.17.7 (#3…
cef707e 
…1421)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/klauspost/compress](https://togithub.com/klauspost/compress)
| `v1.17.6` -> `v1.17.7` |
[![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

###
[`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare
Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

##### What's Changed

- s2: Add AsyncFlush method: Complete the block without flushing by
[@&#8203;Jille](https://togithub.com/Jille) in
[klauspost/compress#927
- s2: Fix literal+repeat exceeds dst crash by
[@&#8203;klauspost](https://togithub.com/klauspost) in
[klauspost/compress#930

**Full Changelog**:
klauspost/compress@v1.17.6...v1.17.7

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMTIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMC4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
XinRanZhAWS pushed a commit to XinRanZhAWS/opentelemetry-collector-contrib that referenced this pull request Mar 13, 2024
@renovate @opentelemetrybot @XinRanZhAWS
fix(deps): update module github.com/klauspost/compress to v1.17.7 (op…
bd0d300 
…en-telemetry#31421)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/klauspost/compress](https://togithub.com/klauspost/compress)
| `v1.17.6` -> `v1.17.7` |
[![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fklauspost%2fcompress/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fklauspost%2fcompress/v1.17.6/v1.17.7?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

###
[`v1.17.7`](https://togithub.com/klauspost/compress/releases/tag/v1.17.7)

[Compare
Source](https://togithub.com/klauspost/compress/compare/v1.17.6...v1.17.7)

##### What's Changed

- s2: Add AsyncFlush method: Complete the block without flushing by
[@&open-telemetry#8203;Jille](https://togithub.com/Jille) in
[klauspost/compress#927
- s2: Fix literal+repeat exceeds dst crash by
[@&open-telemetry#8203;klauspost](https://togithub.com/klauspost) in
[klauspost/compress#930

**Full Changelog**:
klauspost/compress@v1.17.6...v1.17.7

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yMTIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjIyMC4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klauspost klauspost klauspost approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Jille @klauspost