Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add write methods to DataBufferUtils [SPR-15726] #20282

Closed
spring-issuemaster opened this issue Jul 1, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@spring-issuemaster
Copy link
Collaborator

commented Jul 1, 2017

Doron Gold opened SPR-15726 and commented

Although it's usually less frequent than HTTP communication and database access, web applications and micro-services sometimes need to read/write from the filesystem.

When building a web application or a RESTful service with Spring WebFlux, it's best to stay reactive "all the way".
This means that filesystem operations, like any other I/O operation, should be non-blocking.
It is especially important when accessing a mounted filesystem, in such a case operations can be just as slow (and potentially block for a long time) as HTTP operations, since they go over the network.

It would be very helpful to have a high-level utility that facilitates reactive filesystem operations (such as read and write) and abstracts away low-level details such as Netty API.
This would be similar to the high-level way WebClient facilitates HTTP operations.

In a similar fashion to WebClient, such a utility should have an API that uses Mono and Flux, for easily chaining to other operations and for consistency with the rest of the Spring reactive framework.


Issue Links:

  • #20329 Provide control over thread resources used for file I/O in WebFlux
@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 5, 2017

Rossen Stoyanchev commented

Is there any option for non-blocking file I/O in Java? I am not aware of anything. There is AsynchronousFileChannel which submits to a thread pool so async but not non-blocking. That maybe good enough for file I/O and in any case is the most that is available in Java as far as I know.

On top of that in the Spring Framework we have DataBufferUtils that can turn an AsynchronousFileChannel into a Flux so you can read from the file with backpressure. We use that for resource handling we have to read resources and write them to the response. We don't have anything for writing because we haven't needed it yet but it's something we can add.

Would that be okay for you and is there anything more you were looking for?

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 11, 2017

Doron Gold commented

After looking a bit more into the subject I see that, like you say, there is no native way in Java for non-blocking file I/O.

Would you say that in a reactive Spring based web-app/microservice the recommended way to read and write to and from small textual files is using AsynchronousFileChannel with a separate dedicated thread-pool?

Our use-cases so far:

  1. Occasionally reading a configuration file.
  2. Occasionally update a very small file that contains a counter (a number represented in text) - the counter is incremented. This involves a write operation.
@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 12, 2017

Rossen Stoyanchev commented

Yes I would say that's probably the best you can do and since file access seems to more occasional (and hopefully local disk?) in your use case it would probably be sufficient and occasional spike can be managed with the thread pool.

We do need to add support for writing in DataBufferUtils. Arjen Poutsma I'm assigning to you since you know that code best.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 14, 2017

Rossen Stoyanchev commented

Modified title (was: "Analogy of WebClient for filesystem operations")

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 14, 2017

Arjen Poutsma commented

This is now in master, see 83051b0

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 14, 2017

Arjen Poutsma commented

After discussing with Rossen Stoyanchev, we decided to change the return type from Mono<Void> to Flux<DataBuffer>. See c802827

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.