Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support mark() & reset() for TieredFileSegmentInputStream #6624

Closed
TheR1sing3un opened this issue Apr 20, 2023 · 0 comments · Fixed by #6625
Closed

Support mark() & reset() for TieredFileSegmentInputStream #6624

TheR1sing3un opened this issue Apr 20, 2023 · 0 comments · Fixed by #6625
Labels
type/enhancement

Comments

@TheR1sing3un
Copy link
Contributor

In some case, the tiered storage provider will asynchronously read the TieredFileSegmentInputStream, and some upload sdk such as S3 will retry to upload this stream when occurs some error, and the sdk will mark and reset the stream. So it is neccessary to support mark and reset the TieredFileSegmentInputStream.
image

The issue tracker is used for bug reporting purposes ONLY whereas feature request needs to follow the RIP process. To avoid unnecessary duplication, please check whether there is a previous issue before filing a new one.

It is recommended to start a discussion thread in the mailing lists or github discussions in cases of discussing your deployment plan, API clarification, and other non-bug-reporting issues.
We welcome any friendly suggestions, bug fixes, collaboration, and other improvements.

Please ensure that your bug report is clear and self-contained. Otherwise, it would take additional rounds of communication, thus more time, to understand the problem itself.

Generally, fixing an issue goes through the following steps:

  1. Understand the issue reported;
  2. Reproduce the unexpected behavior locally;
  3. Perform root cause analysis to identify the underlying problem;
  4. Create test cases to cover the identified problem;
  5. Work out a solution to rectify the behavior and make the newly created test cases pass;
  6. Make a pull request and go through peer review;

As a result, it would be very helpful yet challenging if you could provide an isolated project reproducing your reported issue. Anyway, please ensure your issue report is informative enough for the community to pick up. At a minimum, include the following hints:

BUG REPORT

  1. Please describe the issue you observed:

  • What did you do (The steps to reproduce)?

  • What is expected to see?

  • What did you see instead?

  1. Please tell us about your environment:

  2. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):

FEATURE REQUEST

  1. Please describe the feature you are requesting.

  2. Provide any additional detail on your proposed use case for this feature.

  3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

  4. If there are some sub-tasks involved, use -[] for each sub-task and create a corresponding issue to map to the sub-task:
@TheR1sing3un TheR1sing3un mentioned this issue Apr 20, 2023
Merged
6 tasks
TheR1sing3un added a commit to TheR1sing3un/rocketmq that referenced this issue May 27, 2023
@TheR1sing3un
perf(tieredstorage): benchmark TieredFileSegmentInputStream pef
ec57da2 
1. benchmark TieredFileSegmentInputStream pef

Closes apache#6624
TheR1sing3un added a commit to TheR1sing3un/rocketmq that referenced this issue May 27, 2023
@TheR1sing3un
feat(tieredstorage): optimized read(byte[], int, int) for TieredFIl…
9e514e9 
…eSegmentInputStream

1. optimized `read(byte[], int, int)` for TieredFIleSegmentInputStream

Closes apache#6624
TheR1sing3un added a commit to TheR1sing3un/rocketmq that referenced this issue May 28, 2023
@TheR1sing3un
fix(tieredstorage): fix a dead cycle in TieredFileSegmentInputStream
71812d5 
1. fix a dead cycle in TieredFileSegmentInputStream.java
2. remove unused JMH
related dependency

Closes apache#6624
zhouxinyu pushed a commit that referenced this issue May 31, 2023
@TheR1sing3un
[ISSUE #6624]Support mark() & reset() for TieredFileSegmentInputStream (
1a9a8b1 
#6625)

* test(tieredstorage): add UT to verify TieredFileSegmentInputStream

1. add UT to verify TieredFileSegmentInputStream

* refactor(tieredstorage): refactor TieredFileSegmentInputStream

1. refactor TieredFileSegmentInputStream

* feat(tieredstorage): support mark&reset TieredFileSegmentInputStream

1. support mark&reset TieredFileSegmentInputStream

* style(tieredstorage): remove commended code in TieredFileSegmentInputStreamTest

1. remove commended code in TieredFileSegmentInputStreamTest

* refactor(tieredstorage): better code placement

1. better code placement

* refactor(tieredstorage): refactor TieredFileSegmentInputStream for better understandability

1. refactor TieredFileSegmentInputStream for better understandability

* refactor(tieredstorage): refactor some code in TieredFileSegmentInputStream

1. refactor some code in TieredFileSegmentInputStream

* refactor(tieredstorage): refactor TieredFileSegmentInputStream

1. refactor TieredFileSegmentInputStream
2. add a
TieredFileSegmentInputStream.Factory to build instance

* refactor(tieredstorage): refactor TieredFileSegmentInputStream related directory structure

1. refactor TieredFileSegmentInputStream related directory structure

* refactor(tieredstorage): delete `commitLogOffsetBuffer` in TieredCommitLogInputStream

1. delete `commitLogOffsetBuffer` in TieredCommitLogInputStream

* perf(tieredstorage): benchmark TieredFileSegmentInputStream pef

1. benchmark TieredFileSegmentInputStream pef

Closes #6624

* feat(tieredstorage): optimized `read(byte[], int, int)` for TieredFIleSegmentInputStream

1. optimized `read(byte[], int, int)` for TieredFIleSegmentInputStream

Closes #6624

* fix(tieredstorage): fix a dead cycle in TieredFileSegmentInputStream

1. fix a dead cycle in TieredFileSegmentInputStream.java
2. remove unused JMH
related dependency

Closes #6624
TheR1sing3un added a commit to TheR1sing3un/rocketmq that referenced this issue May 31, 2023
@TheR1sing3un
fix(tieredstorage): resolve conflicts after rebasing master
fc61063 
1. resolve conflicts after rebasing master

Closes apache#6624
zhouxinyu pushed a commit that referenced this issue Jun 7, 2023
@TheR1sing3un
[ISSUE #6154] Support Amazon S3 backend in TieredStorage (#6495)
450d0d6 
* support s3 backend in tiered storage

* refactor(tieredstorage): Unify all object storage configuration properties

1. Unify all object storage configuration properties

* refactor(tieredstorage): replace some lambda function with more simple expression

1. replace some lambda function with more simple expression

* style(tieredstorage): perfect comments on ChunkMetadata

1. perfect comments on ChunkMetadata

* refactor(tieredstorage): perfect lambda expression

1. perfect lambda expression

* fix(tieredstorage): fix unmatched config attributes in brokerS3.conf

1. fix unmatched config attributes in brokerS3.conf

* feat(tieredstorage): More context in logging output

1. More context in logging output

* fix(tieredstorage): fix wrong concurrently put

1. fix wrong concurrently put

* test(tieredstorage): add UT to verify TieredFileSegmentInputStream

1. add UT to verify TieredFileSegmentInputStream

* refactor(tieredstorage): better code placement

1. better code placement

* refactor(tieredstorage): refactor TieredFileSegmentInputStream for better understandability

1. refactor TieredFileSegmentInputStream for better understandability

* feat(tieredstorage): support `reset` of TieredFileSegmentInputStream

1. support `reset` of TieredFileSegmentInputStream

* fix(tieredstorage): fix wrong position when failed in `S3FileSegment#commit0`

1. fix wrong position when failed in `S3FileSegment#commit0`

* fix(tieredstorage): fix still have upload buffer when already seal the segment

1. fix still have upload buffer when already seal the segment

* test(tieredstorage): fix wrong assertion

1. fix wrong assertion

* feat(tieredstorage): support switch to enable merge chunks into segment

1. support switch to enable merge chunks into segment

* feat(tieredstorage): add more debug log in TieredMessageStore

1. add more debug log in TieredMessageStore

* style(tieredstorage): use rmq code style

1. use rmq code style

* feat(tieredstorage): add metrics for S3 provider

1. add metrics for S3 provider

* fix(tieredstorage): resolve conflicts after rebasing master

1. resolve conflicts after rebasing master

Closes #6624

* style(tieredstorage): change log level

1. change log level

Closes #6154

* build(controller): build tieredstorage with bazel

1. build tieredstorage with bazel

* build(controller): build tieredstorage with bazel

1. build tieredstorage with bazel

* style(tieredstorage): change log level

1. change log level

Closes #6154

* test(tieredstorage): ignore tests about S3Mock

1. ignore tests about S3Mock

* test(tieredstorage): ignore tests about S3Mock

1. ignore tests about S3Mock
oopooa pushed a commit to oopooa/rocketmq-reading that referenced this issue Apr 25, 2024
@TheR1sing3un
fix(tieredstorage): resolve conflicts after rebasing master
2f6add4 
1. resolve conflicts after rebasing master

Closes apache/rocketmq#6624
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ISSUE #6624]Support mark() & reset() for TieredFileSegmentInputStream TheR1sing3un/rocketmq
2 participants
@ni-ze @TheR1sing3un