ReadByte() from HttpWebRequest is slow.

[mderoy]

Steps to Reproduce

1. Setup an Http Server on localhost. I setup http-server which can be easily installed with nodejs via npm " npm install http-server -g" see https://github.com/indexzero/http-server
2. cd into a folder with a file that is 300KB called bigfile2.txt
3. Start the webserver 'http-server . -a localhost -p 80'
4. Run my sample code, see the time it takes to download the 300kb a single byte at a time

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp3
{
	class Program
	{
		static void Main(string[] args)
		{
			string streamTestURL = "http://localhost/bigfile2.txt";

			// create HTTP request
			HttpWebRequest req = (HttpWebRequest)WebRequest.Create(streamTestURL);
			// get response
			WebResponse resp = req.GetResponse();

			System.IO.Stream imagestream = resp.GetResponseStream();
			var stopWatch = new System.Diagnostics.Stopwatch();
			stopWatch.Start();
			int a = 0;
			while (a != -1)
			{
				a = imagestream.ReadByte();
			}

			Console.WriteLine("Time elapsed: " + stopWatch.Elapsed);
		}
	}
}

Current Behavior

Downloading 300KB takes about 16.5 seconds from localhost

Expected Behavior

Downloading 300KB with .NET framework downloads the file from localhost in less than 1/10th of a second

On which platforms did you notice this

[ ] macOS
[ ] Linux
[ x ] Windows

Version Used:
Recent Master

Stacktrace

N/A

[EgorBo]

My results:
.NET 4.7.1: 00:00:00.15
mono 5.13: 00:00:00.62

(master/7f4a4965899)

[mderoy]

@EgorBo did you test this on windows?

[EgorBo]

@mderoy yeah, Windows 10 16299.248 (fall creators update)
300kb file (exact size), Release x64

UPD: confirmed - on bigger files the difference is huge (a second for .NET and more than 10 minutes for mono for 20mb txt file).

[migueldeicaza]

The async-based web stack made a big difference for me, about 25 times faster, this was using a master build from today, but I only tested on Mac.

[baulig]

When you're reading the response one byte at a time, each single byte needs to make one of more async operations internally.

With my new code, your example calls async Task<int> ReadAsync (byte[] buffer, int offset, int count, CancellationToken cancellationToken) in both WebResponseStream and FixedSizeReadStream and it will be wrapped in another task by HttpWebRequest.RunWithTimeout. Then, once you hit FixedSizeReadStream, it will actually read the next byte from the already buffed content.

The problem is simply that before the implementation can figure out that the entire buffer you're requesting to read is already buffered, it needs to go through a layer or intermediate checks. It needs to prevent concurrent async calls, setup a timeout cancellation and check for gzip or chunked encoding.

[baulig]

Is there any specific reason why you are reading the response one byte at a time?

To read a 300k file that way, you're doing over a million async requests. Reading even a 256 byte buffer at a time will reduce the time to just milliseconds.

If you truly need to read one byte at a time, then something you could try is to wrap the stream that's returned from GetResponseStream() with some kind of a buffering stream.

[joncham]

This came from a Unity customer. If the underlying issue can be mitigated (say by reading 256 bytes) we can respond and close.

[migueldeicaza]

yes, it can be mitigated with using buffers instead of reading one byte at a time.

[zaneclaes]

@migueldeicaza I stumbled upon this thread in attempting to debug the same problem in Unity, however I'm using ReadAsync with a large buffer and still experiencing the same problems. This code takes ~8 seconds to run on a 6MB test asset (no matter what chunkSize I use):

private async Task DownloadFileChunked(string url, int chunkSize) {
  Debug.Log($"Downloading {url}");
  Stopwatch sw = new Stopwatch();
  sw.Start();

  HttpWebRequest req = (HttpWebRequest) WebRequest.Create(url);
  HttpWebResponse resp = (HttpWebResponse) req.GetResponse();

  int tr = 0;
  using (Stream stream = resp.GetResponseStream()) {
    int r = 0;
    byte[] buffer = new byte[chunkSize];
    while ((r = await stream.ReadAsync(buffer, 0, chunkSize)) != 0) {
      // Debug.Log($"Read {r}.");
      tr += r;
    }
  }
  Debug.Log($"[{sw.ElapsedMilliseconds} ms] Finished {tr} bytes from {url}.");
}

Whereas this code runs in less than 1 second:

private async Task DownloadFileWhole(string url) {
  Debug.Log($"Downloading {url}");
  Stopwatch sw = new Stopwatch();
  sw.Start();
  HttpWebRequest req = (HttpWebRequest) WebRequest.Create(url);
  HttpWebResponse resp = (HttpWebResponse) req.GetResponse();
  string res = await new StreamReader(resp.GetResponseStream()).ReadToEndAsync();
  Debug.Log($"[{sw.ElapsedMilliseconds} ms] Finished {res.Length} characters from {url}.");
}

If I uncomment the line in the first function which outputs the "read" size, I note that the vast majority of the reads are 17408 (aka, exactly 17 kb), which may point at some sort of internal buffer limit (?).