cabal update fails to download package list #174

Closed
bos opened this Issue May 24, 2012 · 2 comments

Projects

None yet

1 participant

@bos
Contributor
bos commented May 24, 2012

(Imported from Trac #181, reported by guest on 2007-11-20)

cabal update fails to download package list

>cabal update
Downloading package list from server 'http://hackage.haskell.org/packages/archive'
cabal: user error (Codec.Compression.Zlib: premature end of compressed stream)
>cabal install bzlib
cabal: Data.ByteString.Lazy.index: index too large: 0
`WinXp?`, GHC 6.8, Cabal 1.3.x
@bos
Contributor
bos commented May 24, 2012

(Imported comment by guest on 2007-11-20)

I think I've tracked this down.

The failure's happening at Hackage/Update.hs. The 00-index.tar.gz in cabal/packages/hackage.haskell.org/ is around 180KB. If I change updateRepo as follows:

updateRepo cfg repo =
    do printf "Downloading package list from server '%s'\n" (repoURL repo)
       indexPath <- downloadIndex cfg repo
       file <- BS.readFile indexPath
       printf "length is %d" $ BS.length file
       -- should be the raw bits
       BS.writeFile "c:/foo.tar.gz" file
       BS.writeFile (dropExtension indexPath) (gunzip file)
       return ()
The foo.tar.gz file has 105 bytes. The 106th byte of 00-index.tar.gz is a ^Z, which I think is getting interpreted as the Windows EOF character. If you change "Data.`ByteString?`.Lazy.Char8" to "Data.`ByteString?`.Lazy" I think you get the right semantics. Here's a patch. I'm not happy about the c2w and w2c calls everywhere, but it works on Windows now.
  • Hackage/Index.hs

    diff -r 225b3427562f Hackage/Index.hs
        <table cellspacing="0" class="inline" summary="Differences">
              <colgroup><col class="lineno" /><col class="lineno" /><col class="content" /></colgroup>
              <thead>
                <tr>
                  <th title="File a/Hackage/Index.hs Sat Dec 15 14:19:33 2007 -0500">
                      a
                  </th>
                  <th title="File b/Hackage/Index.hs Sat Dec 15 14:48:46 2007 -0500">
                      b
                  </th>
                  <th> </th>
                </tr>
              </thead>
            <tbody class="unmod">
                  <tr>
                          <th>18</th><th>18</th><td class="l"><span></span> </td>
                  </tr><tr>
                          <th>19</th><th>19</th><td class="l"><span>import Prelude hiding (catch)</span> </td>
                  </tr><tr>
                          <th>20</th><th>20</th><td class="l"><span>import Control.Exception (catch, Exception(IOException))</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>21</th><th> </th><td class="l"><span>import qualified Data.ByteString.Lazy.Char8 as BS</span> </td>
                      </tr><tr>
                        <th>22</th><th> </th><td class="l"><span>import Data.ByteString.Lazy.Char8 (ByteString)</span> </td>
                      </tr>
                      <tr>
                        <th> </th><th>21</th><td class="r"><span>import qualified Data.ByteString.Lazy as BS</span> </td>
                      </tr><tr>
                        <th> </th><th>22</th><td class="r"><span>import Data.ByteString.Internal (w2c)</span> </td>
                      </tr><tr class="last">
                        <th> </th><th>23</th><td class="r"><span>import Data.ByteString.Lazy (ByteString)</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>23</th><th>24</th><td class="l"><span>import System.FilePath ((&lt;/&gt;), takeExtension, splitDirectories, normalise)</span> </td>
                  </tr><tr>
                          <th>24</th><th>25</th><td class="l"><span>import System.IO (hPutStrLn, stderr)</span> </td>
                  </tr><tr>
                          <th>25</th><th>26</th><td class="l"><span>import System.IO.Error (isDoesNotExistError)</span> </td>
                  </tr>
            </tbody>
              <tbody class="skipped">
                <tr>
                  <th><a href="#L48">…</a></th>
                  <th><a href="#L49">…</a></th>
                  <td> </td>
                </tr>
              </tbody>
            <tbody class="unmod">
                  <tr>
                          <th>48</th><th>49</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp;if takeExtension (tarFileName hdr) == ".cabal"</span> </td>
                  </tr><tr>
                          <th>49</th><th>50</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;then case splitDirectories (normalise (tarFileName hdr)) of</span> </td>
                  </tr><tr>
                          <th>50</th><th>51</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [pkgname,vers,_] -&gt;</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>51</th><th> </th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; let descr = case parsePackageDescription (<del>BS.unpack content</del>) of</span> </td>
                      </tr>
                      <tr class="last">
                        <th> </th><th>52</th><td class="r"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; let descr = case parsePackageDescription (<ins>map w2c (BS.unpack content)</ins>) of</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>52</th><th>53</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ParseOk _ d -&gt; d</span> </td>
                  </tr><tr>
                          <th>53</th><th>54</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; _&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;-&gt; error $ "Couldn't read cabal file "</span> </td>
                  </tr><tr>
                          <th>54</th><th>55</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ++ show (tarFileName hdr)</span> </td>
                  </tr>
            </tbody>
        </table>
      </li>
      <li class="entry">
        <h2>
          <a>Hackage/Tar.hs</a>
        </h2>
        <pre>diff -r 225b3427562f Hackage/Tar.hs</pre>
        <table cellspacing="0" class="inline" summary="Differences">
              <colgroup><col class="lineno" /><col class="lineno" /><col class="content" /></colgroup>
              <thead>
                <tr>
                  <th title="File a/Hackage/Tar.hs Sat Dec 15 14:19:33 2007 -0500">
                      a
                  </th>
                  <th title="File b/Hackage/Tar.hs Sat Dec 15 14:48:46 2007 -0500">
                      b
                  </th>
                  <th> </th>
                </tr>
              </thead>
            <tbody class="unmod">
                  <tr>
                          <th>3</th><th>3</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;readTarArchive, extractTarArchive, </span> </td>
                  </tr><tr>
                          <th>4</th><th>4</th><td class="l"><span>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;extractTarGzFile, gunzip) where</span> </td>
                  </tr><tr>
                          <th>5</th><th>5</th><td class="l"><span></span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>6</th><th> </th><td class="l"><span>import qualified Data.ByteString.Lazy.Char8 as BS</span> </td>
                      </tr><tr>
                        <th>7</th><th> </th><td class="l"><span>import Data.ByteString.Lazy.Char8 (ByteString)</span> </td>
                      </tr>
                      <tr>
                        <th> </th><th>6</th><td class="r"><span>import qualified Data.ByteString.Lazy as BS</span> </td>
                      </tr><tr>
                        <th> </th><th>7</th><td class="r"><span>import Data.ByteString.Internal(c2w,w2c)</span> </td>
                      </tr><tr class="last">
                        <th> </th><th>8</th><td class="r"><span>import Data.ByteString.Lazy (ByteString)</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>8</th><th>9</th><td class="l"><span>import Data.Bits ((.&amp;.))</span> </td>
                  </tr><tr>
                          <th>9</th><th>10</th><td class="l"><span>import Data.Char (ord)</span> </td>
                  </tr><tr>
                          <th>10</th><th>11</th><td class="l"><span>import Data.Int (Int8, Int64)</span> </td>
                  </tr>
            </tbody>
              <tbody class="skipped">
                <tr>
                  <th><a href="#L134">…</a></th>
                  <th><a href="#L135">…</a></th>
                  <td> </td>
                </tr>
              </tbody>
            <tbody class="unmod">
                  <tr>
                          <th>134</th><th>135</th><td class="l"><span>checkChkSum hdr s = s == chkSum hdr' || s == signedChkSum hdr'</span> </td>
                  </tr><tr>
                          <th>135</th><th>136</th><td class="l"><span>&nbsp; where </span> </td>
                  </tr><tr>
                          <th>136</th><th>137</th><td class="l"><span>&nbsp; &nbsp; -- replace the checksum with spaces</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>137</th><th> </th><td class="l"><span>&nbsp; &nbsp; hdr' = BS.concat [BS.take 148 hdr, BS.replicate 8 <del>' '</del>, BS.drop 156 hdr]</span> </td>
                      </tr>
                      <tr class="last">
                        <th> </th><th>138</th><td class="r"><span>&nbsp; &nbsp; hdr' = BS.concat [BS.take 148 hdr, BS.replicate 8 <ins>(c2w ' ')</ins>, BS.drop 156 hdr]</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>138</th><th>139</th><td class="l"><span>&nbsp; &nbsp; -- tar.info says that Sun tar is buggy and </span> </td>
                  </tr><tr>
                          <th>139</th><th>140</th><td class="l"><span>&nbsp; &nbsp; -- calculates the checksum using signed chars</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>140</th><th> </th><td class="l"><span>&nbsp; &nbsp; chkSum = BS.foldl' (\x y -&gt; x + ord <del>y</del>) 0</span> </td>
                      </tr><tr>
                        <th>141</th><th> </th><td class="l"><span>&nbsp; &nbsp; signedChkSum = BS.foldl' (\x y -&gt; x + (ordSigned <del>y</del>)) 0</span> </td>
                      </tr>
                      <tr>
                        <th> </th><th>141</th><td class="r"><span>&nbsp; &nbsp; chkSum = BS.foldl' (\x y -&gt; x + ord <ins>(w2c y)</ins>) 0</span> </td>
                      </tr><tr class="last">
                        <th> </th><th>142</th><td class="r"><span>&nbsp; &nbsp; signedChkSum = BS.foldl' (\x y -&gt; x + (ordSigned <ins>(w2c y)</ins>)) 0</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>142</th><th>143</th><td class="l"><span></span> </td>
                  </tr><tr>
                          <th>143</th><th>144</th><td class="l"><span>ordSigned :: Char -&gt; Int</span> </td>
                  </tr><tr>
                          <th>144</th><th>145</th><td class="l"><span>ordSigned c = fromIntegral (fromIntegral (ord c) :: Int8)</span> </td>
                  </tr>
            </tbody>
              <tbody class="skipped">
                <tr>
                  <th><a href="#L156">…</a></th>
                  <th><a href="#L157">…</a></th>
                  <td> </td>
                </tr>
              </tbody>
            <tbody class="unmod">
                  <tr>
                          <th>156</th><th>157</th><td class="l"><span>getBytes off len = BS.take len . BS.drop off</span> </td>
                  </tr><tr>
                          <th>157</th><th>158</th><td class="l"><span></span> </td>
                  </tr><tr>
                          <th>158</th><th>159</th><td class="l"><span>getByte :: Int64 -&gt; ByteString -&gt; Char</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>159</th><th> </th><td class="l"><span>getByte off bs = <del></del>BS.index bs off</span> </td>
                      </tr>
                      <tr class="last">
                        <th> </th><th>160</th><td class="r"><span>getByte off bs = <ins>w2c $ </ins>BS.index bs off</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>160</th><th>161</th><td class="l"><span></span> </td>
                  </tr><tr>
                          <th>161</th><th>162</th><td class="l"><span>getString :: Int64 -&gt; Int64 -&gt; ByteString -&gt; String</span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>162</th><th> </th><td class="l"><span>getString off len = <del>BS.unpack . BS.takeWhile (/='\0') . getBytes off len</del></span> </td>
                      </tr>
                      <tr class="last">
                        <th> </th><th>163</th><td class="r"><span>getString off len = <ins>\x -&gt; map w2c ((BS.unpack . BS.takeWhile (/= (c2w '\0')) . getBytes off len) x)</ins></span> </td>
                      </tr>
            </tbody>
        </table>
      </li>
      <li class="entry">
        <h2>
          <a>Hackage/Update.hs</a>
        </h2>
        <pre>diff -r 225b3427562f Hackage/Update.hs</pre>
        <table cellspacing="0" class="inline" summary="Differences">
              <colgroup><col class="lineno" /><col class="lineno" /><col class="content" /></colgroup>
              <thead>
                <tr>
                  <th title="File a/Hackage/Update.hs Sat Dec 15 14:19:33 2007 -0500">
                      a
                  </th>
                  <th title="File b/Hackage/Update.hs Sat Dec 15 14:48:46 2007 -0500">
                      b
                  </th>
                  <th> </th>
                </tr>
              </thead>
            <tbody class="unmod">
                  <tr>
                          <th>18</th><th>18</th><td class="l"><span>import Hackage.Fetch</span> </td>
                  </tr><tr>
                          <th>19</th><th>19</th><td class="l"><span>import Hackage.Tar</span> </td>
                  </tr><tr>
                          <th>20</th><th>20</th><td class="l"><span></span> </td>
                  </tr>
            </tbody><tbody class="mod">
                      <tr class="first">
                        <th>21</th><th> </th><td class="l"><span>import qualified Data.ByteString.Lazy<del>.Char8</del> as BS</span> </td>
                      </tr>
                      <tr class="last">
                        <th> </th><th>21</th><td class="r"><span>import qualified Data.ByteString.Lazy<ins></ins> as BS</span> </td>
                      </tr>
            </tbody><tbody class="unmod">
                  <tr>
                          <th>22</th><th>22</th><td class="l"><span>import System.FilePath (dropExtension)</span> </td>
                  </tr><tr>
                          <th>23</th><th>23</th><td class="l"><span>import Text.Printf</span> </td>
                  </tr>
            </tbody>
        </table>
      </li>
    
-- greg@…
@bos
Contributor
bos commented May 24, 2012

(Imported comment by @dcoutts on 2007-12-15)

Excellent! Great work, I'm glad you worked out what it was.

As it happens we've already made the change you suggest because I found something similar with unpacking files on windows and them getting corrupt due to line ending conversions. So in the development version we're now using the non-Char8 versions throughout.

@bos bos closed this May 24, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment