Support for large DBF/SHP files

[mapserver-bot]

Reporter: jmckenna
Date: 2010/08/13 - 17:12
Trac URL: http://trac.osgeo.org/mapserver/ticket/3514
One of the requirement's of this year's benchmarking exercise is to display a large shapefile (DBF is 2.9GB, SHP is 1.3 GB). The file's name is 'buildings.shp'. The problem is that with native MapServer the file is not displaying. If I change to an OGR connection the file does display. Other server teams can display this same shapefile (in fact 2 days ago GeoServer had to be modified to display this same large shapefile...'use longs instead of ints for dbase offsets'). MapServer must also be hitting some kind of limit with this file.

I have created a small mapfile on the Linux and Windows benchmark machines (/benchmarking/mapserver/buildings-test.map). We can provide full access to these machines if you wish.

Also here are working getmap requests showing our problem:

• geoserver: http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=3.023023223877,41.949409484865,3.0487724304201,41.975158691409&WIDTH=600&HEIGHT=600
• mapserve (native): http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&LAYERS=vector_benchmarking_2010&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=3.023023223877,41.949409484865,3.0487724304201,41.975158691409&WIDTH=600&HEIGHT=600

[mapserver-bot]

Author: warmerdam
Date: 2010/08/13 - 17:31
The same issue was encountered in shapelib, and the solution was too use a "large file API" for access to the DBF file instead of the traditional '32bit signed' stdio. There may also need to be some care with use of unsigned longs instead of signed longs for offset calculations.

I could take on this ticket if Daniel wants it addressed as part of our maintenance arrangement.

[mapserver-bot]

Author: pramsey
Date: 2010/08/13 - 17:41
The "hack fix" approach is available here, if you are in a hurry and not committing back.

http://trac.osgeo.org/postgis/changeset/5787

[mapserver-bot]

Author: warmerdam
Date: 2010/08/13 - 17:51
I would note this (Paul's referenced patch) is only likely to help on 64bit systems with long and int are not the same.

[mapserver-bot]

Author: dmorissette
Date: 2010/08/16 - 17:12
Frank, yes please go ahead. Note however that since this is more than just a trivial fix it should happen only in 6.0 and cannot be backported to 5.6 unless the PSC decides to make an exception.

[mapserver-bot]

Author: warmerdam
Date: 2010/08/19 - 06:01
My usual approach to large file support is to "hook" the IO api. On unix there are a few options as seen in gdal/port/cpl_vsil_unix_stdio_64.cpp while on windows I use the native win32 file io api (CreateFile(), etc). However, this is a fairly heavy weight approach. Digging a bit deeper, for instance in http://www.suse.de/~aj/linux_lfs.html it seems there are easier alternatives.

I have confirmed that on 32bit linux defining _FILE_OFFSET_BITS=64 allows writing large files. But it does not alter the "long" argument to fseek(). I will continue to dig...

[mapserver-bot]

Author: warmerdam
Date: 2010/08/19 - 07:08
Preliminary incompletely tested fix in trunk (63101db (r10462), 1bddcb4 (r10463)). I have not confirmed mapserver itself working with a >2GB file. The fix should work on VStudio 2005 or later, newish 32bit unix/linux and 64bit unix/linux.

It would be fairly involved to test so I'm going to "assume" it works for now.

[mapserver-bot]

Author: jmckenna
Date: 2010/08/19 - 13:05
I have confirmed this fix with the problem 2.9GB DBF file, on Windows x64. Great work Frank!!!! Thank you so much. I'll ask Mike to rebuild MapServer on Linux to test and I will report back.