Efficient std.conv.to conversions

[dlangBugzillaToGithub]

tomeksowi reported this on 2011-02-08T12:47:58Z

Transfered from https://issues.dlang.org/show_bug.cgi?id=5548

CC List

• andrei (@andralex)
• blah38621
• jack (@JackStouffer)
• jrdemail2000-dlang
• lovelydear
• schveiguy (@schveiguy)

Description

Currently most to!T conversions allocate behind the scenes. I propose that the conversions where T is dynamically sized (like an array) get a speedy overload of the form:

void to(T, S, O)(S source, O output) if (isOutputRange!(O, T));

Example: to!string(9234820934, appender);

For further investigation: consider assuming the output range to be buffered.

[dlangBugzillaToGithub]

schveiguy (@schveiguy) commented on 2011-02-08T12:52:10Z

Do we not have the functionality for this (at least for strings) in std.format?

Are there use cases for this beyond strings?

[dlangBugzillaToGithub]

k.hara.pg commented on 2012-06-14T02:33:25Z

I agree with Steven. If you want to represent objects with output range of characters, you can use std.format.formatValue family directly.

[dlangBugzillaToGithub]

blah38621 commented on 2013-11-01T08:26:19Z

After a bit of testing, std.format.fomatValue is not a valid alternative to having std.conv.to provide overloads that accept an output buffer. The reason for this is simple, the following code is 3x slower than simply using to!string(int):


auto toStr = benchmark!(() {
	import std.format;
	import std.range : Appender;
	
	auto ret = Appender!string();
	auto fmt = FormatSpec!char("%s");
	ret.reserve(4096);
	for (auto i = 0; i < ObjectCount * 11; i++)
	{
		ret.formatValue(i, fmt);
		ret.clear();
	}
})(1);
	 
writefln("Took %s ms (%s) to serialize 100k SimpleObjects with an average payload of %s bytes (%s).", res[0].msecs, toStr[0].msecs, cast(real)totalPayload / ObjectCount, totalPayload);


In my tests where ObjectCount was 100k, it takes 400ms for to!string(int) to create all the strings, and 1100ms for formatValue to do the same. formattedWrite is even worse, 1500ms. In my current implementation of a dynamic JSON (de)serializer, more than half of my time is eaten up by converting integers to strings when performing deserialization. I use a pre-allocated output range as the destination, so I know I'm not doing any allocations within my code.

[dlangBugzillaToGithub]

jack (@JackStouffer) commented on 2017-02-02T16:03:27Z

Definitely agree that this should exist. Currently in the array overloads, there are a lot of uses of functions that use appender internally and then return the managed array with app.data(), only to be appended to another appender in the parent function. Having OutputRange overloads would represent a significant performance boost.

Once https://github.com/dlang/phobos/pull/5018 is pulled, I'll start working on this.

BTW, in the mean time, converting to string can be used with OutputRanges via std.range.put and std.conv.toChars.