SNAP has problems serving large files (jpegs)

[ChasTheSpaz]

I am developing an image system in Haskell, using the Threepenny webserver. Testing of the prototype system showed problems presenting large jpeg images, from incomplete loading to totally invalidated loading. I consulted with Heinrich Apfelmus about this and he said that SNAP is part of his code that serves the file. I used a small test program directly in SNAP and can consistently produce errors on large jpeg files.

module Main where

import Snap.Core           (Snap, route)
import Snap.Http.Server    (quickHttpServe)
import Snap.Util.FileServe
import Data.ByteString.Char8

site :: Snap ()
site = route
  [ (pack "/files",   serveDirectory                            "static")
  , (pack "/simple",  serveDirectoryWith simpleDirectoryConfig  "static")
  , (pack "/default", serveDirectoryWith defaultDirectoryConfig "static")
  , (pack "/fancy",   serveDirectoryWith fancyDirectoryConfig   "static")
  , (pack "/hy",      serveFile                                 "static/hy.jpg")
  , (pack "/ostriches", serveFile                               "static/ostriches.jpg")
  , (pack "/sunflower", serveFile                               "static/sunflower150x113.png")
  , (pack "/hesperid",  serveFile                               "static/hesperid.jpeg")
  , (pack "/E",         serveFile                               "static/E.png")
  , (pack "/3-14",      serveFile                               "static/3-14.jpg")
  , (pack "/3-20",      serveFile                               "static/3-20.jpg")
  , (pack "/53",        serveFile                               "static/53.jpeg")
  ]
 
main :: IO ()
main = quickHttpServe site

None of the files listed can be attached for reasons known only to github.
Comments: file hy.jpg is 13mb, ostriches.jpg in 7.7 mb, hesperid.jpeg is 820 kb, the remaining are
much smaller files that served up OK.

This issue is also snapframework/snap#194 slightly modified text.

[imalsogreg]

I can confirm this on macOS. serveDirectory returns strangely truncated files. Different truncations for different requests for the same file.

> curl localhost:8000/out.js > tmp.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7032k  100 7032k    0     0   371M      0 --:--:-- --:--:-- --:--:--  381M

> tail tmp.js
            var aD = ((aC + 224) | 0);
            i.u8[(j + 0)] = (aD & 255);
            var aE = (k >>> 6);
            var aF = (aE & 63);
            var aG = ((aF + 128) | 0);
            var aH = (aG & 255);
            var aI;
            var aJ;
            aI = i;
            aJ%

> wc tmp.js
  340572 1005833 7200868 tmp.js

Minutes later,

> curl localhost:8000/out.js > tmp.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7032k  100 7032k    0     0   348M      0 --:--:-- --:--:-- --:--:--  361M

> tail tmp.js
            var O = N;
            var P = ((O - 56320) | 0);
            var Q = j;
            var R = ((Q - 55296) | 0);
            var S = (R << 10);
            var T = ((S + P) | 0);
            var U = ((T + 65536) | 0);
            var V = (U >> 18);
            var W = ((V + 240) | 0);
            h.u8[(i + 0%

> wc tmp.js
  344159  981752 7200868 tmp.js

It seems strange that ls -l tmp.js always returns the size 7200868. The same size as the real file being served.

The same snap code compiled on nixos in a VM on the macos machine serves the file correctly.

[mightybyte]

Pinging @gregorycollins

[gregorycollins]

I think something is wrong with the sendfile backend on OSX -- are all of these bug reports happening on Mac?

[gregorycollins]

I turned off sendfile support on OSX. Please try the latest snap-server and reopen this if it's still happening.