Add PathStreamMessage that publishes the contents of a Path
#3344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Motivation:
It must be nice to create `StreamMessage` from a `File` or `Path`.
Modifications:
- Add `PathStreamMessage` that reads bytes from a `Path` and publishes with backpressure
- Add useful factories to `StreamMessage`
Result:
You can now create a `StreamMessage` from a `Path` or `File`.
```java
Path path = Paths.get("...");
StreamMessage<HttpData> publisher = StreamMessage.of(path);
```
ikhoon
changed the title
PathStreamMessage that publishes a PathPathStreamMessage that publishes the contents of a Path
Codecov Report
@@ Coverage Diff @@
## master #3344 +/- ##
============================================
+ Coverage 74.19% 74.26% +0.06%
- Complexity 13293 13324 +31
============================================
Files 1159 1160 +1
Lines 50740 50918 +178
Branches 6504 6530 +26
============================================
+ Hits 37646 37812 +166
- Misses 9775 9782 +7
- Partials 3319 3324 +5
Continue to review full report at Codecov.
|
minwoox
reviewed
Feb 15, 2021
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/StreamMessage.java
Outdated
Show resolved
Hide resolved
minwoox
approved these changes
Feb 17, 2021
core/src/main/java/com/linecorp/armeria/common/stream/StreamMessage.java
Show resolved
Hide resolved
trustin
reviewed
Feb 26, 2021
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/PathStreamMessage.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/common/stream/StreamMessage.java
Outdated
Show resolved
Hide resolved
core/src/test/java/com/linecorp/armeria/common/stream/PathStreamMessageTest.java
Outdated
Show resolved
Hide resolved
trustin
reviewed
Mar 7, 2021
| @@ -0,0 +1,368 @@ | |||
| /* | |||
| * Copyright 2020 LINE Corporation | |||
|
|
||
| private static final Logger logger = LoggerFactory.getLogger(PathStreamMessage.class); | ||
|
|
||
| static final int DEFAULT_FILE_BUFFER_SIZE = 4096; |
There was a problem hiding this comment.
Maybe 8192? 4K may be too small for modern OSes.
| byteBuf.release(); | ||
| maybeCloseFileChannel(); | ||
| } else { | ||
| if (result > -1) { |
Comment on lines
309
to
316
| byteBuf.writerIndex(result); | ||
| final HttpData data; | ||
| if (withPooledObjects) { | ||
| data = HttpData.wrap(byteBuf); | ||
| } else { | ||
| data = HttpData.wrap(ByteBufUtil.getBytes(byteBuf)); | ||
| byteBuf.release(); | ||
| } |
There was a problem hiding this comment.
Could we do this without updating reader/writerIndex?
There was a problem hiding this comment.
IIUC, we need to call byteBuf.writerIndex(result) because the indexes of byteBuf are not changed by ByteBuffer.
https://github.com/netty/netty/blob/e5951d46fc89db507ba7d2968d2ede26378f0b04/buffer/src/main/java/io/netty/buffer/ByteBuf.java#L2292-L2294
* Exposes this buffer's sub-region as an NIO {@link ByteBuffer}. The returned buffer * either share or contains the copied content of this buffer, while changing the position * and limit of the returned NIO buffer does not affect the indexes and marks of this buffer.
Comment on lines
342
to
343
| } catch (IOException ignored) { | ||
| } |
| @ArgumentsSource(SubscriptionOptionsProvider.class) | ||
| @ParameterizedTest | ||
| void readFile(SubscriptionOption[] options) { | ||
| final Path path = Paths.get("src/test/resources/com/linecorp/armeria/common/stream/test.txt"); |
There was a problem hiding this comment.
Could we test a file larger than the buffer size as well as a small one?
trustin
approved these changes
Mar 8, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
It must be nice to create
StreamMessagefrom aFileorPath.Modifications:
PathStreamMessagethat reads bytes from aPathand publishes with backpressureStreamMessageResult:
You can now create a
StreamMessagefrom aPathorFile.