Inefficient write! macro with vector

[DawidPi]

write! macro, that uses vector to write it's elements into a file is inefficient, as it make write() system call for each letter stored in vector of chars. It could be implemented in a way, that only one write system call is made with all the data, that is to be written.

source code, that shows the issue can be seen with following source code:

extern crate rand;

use rand::Rng;
use std::io::Write;
use std::fs::OpenOptions;

fn generate_character() -> char {
    rand::thread_rng().gen_ascii_chars().next().expect("Could not get character")
}

fn main() {
    let mut vector = Vec::new();

    for _ in 0..10000{
        vector.push(generate_character());
    }

    let mut file = OpenOptions::new().create(true).write(true).open("testfile.txt").unwrap();

    write!(file, "{:?}", vector).expect("Could not write to file");
    file.sync_all().expect("Could not sync all");

    let string : String = vector.iter().collect();
    file.write_all(string.as_bytes()).expect("Could not sync all");
    file.sync_all().expect("Could not sync all");

}

(there is a wrong error log if writing fails but it's not important)

when progam is run with strace we can see, that write! under the hood issues many many system calls, while write_all method uses just one system call. One system call is often more efficient and in some cases even much more efficient.

Attaching logs from the strace and sample project with given source code.

Is there a possibility to fix this issue? I am no Rust expert at all and with macros I know very little, but this certainly is an issue. I see this a some major flaw

strace.log
inefficient write macro.zip

[sfackler]

Wrap your File in a BufWriter: https://doc.rust-lang.org/std/io/struct.BufWriter.html.

[DawidPi]

I don't find this a proper solution. It's just as fine as call to the write_all method, but still It's just a workaround. As a user I would expect that using standard library should be quite fast and that there is no need to do any workaround.

From the api point of view I am giving the macro both a file and full vector of data to be written. Also I want write syscall to be called when I call write! macro(I don't want any buffering or other magic as buffered data is already provided).

As macro has all the data it needs to make efficient writing (with single write syscall) it should do so. Also there is no drawback to the solution with many syscalls. It would be just much faster.

Is it possible to implement this macro in a way, that it uses 1 syscall?

[sfackler]

As macro has all the data it needs to make efficient writing (with single write syscall) it should do so.

How is this going to be implemented? Is it going to internally buffer the output into a vector and then write that out? What's the cost to allocating that buffer in every write call? What happens when you're using write! in a no_std context where you can't allocate memory? write! can be used with any writer, not just unbuffered files. How much is the performance of writing to a Vec<u8> going to be impacted by this intermediary buffer layer?

[hanna-kruppe]

You're not even comparing two different implementations of the same operation. Formatting a Vec<char> with {:?} does something very different from outputting the UTF-8 encoded String: It outputs (for example) ['5', '6', 'n', 'O', 'Y', 'm', 'u', '5', 'l', 'o'] instead of 56nOYmu5lo.

Furthermore, even if it did output the latter, Vec<char> is not UTF-8 encoded internally, so there's no way to just output its contents as UTF-8 in one syscall without copying/transcoding first (compare this with String which is stored as UTF-8). At most it could output UTF-32, but this is incompatible with the std::fmt infrastructure used by write! (std::fmt can only output UTF-8).

[DawidPi]

Ahh, now I think I got it, why It's like that, didn't know about this additional formatting in this case, sorry for that. If that is the case, then would this: ''' use std::io::Write; fn main() { let x = "test".to_string(); let mut output = std::fs::OpenOptions::new().create(true).write(true).open("file").unwrap(); write!(output, "{}", x).unwrap(); } ''' Issue one system call? According to what you say, string should contain data that is ready to be just flushed to the file, there should be also no additional formatting, so it can just issue write giving it's buffer as an argument (no additional formatting and allocations) right? I will check it tomorrow, but if it's issuing one call, then It's ok.

[DawidPi]

Ok I checked it and it indeed issues one system call. I will close the issue, then. Sorry for that.