-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ticket #5272 - Speed improvement for OGR shapefile provider #24
Conversation
It is possible improve the shapefile reader avoiding unnecessary memcpy's and calloc's and replacing the CPLAtof by super-fast OGRFastAtof. It gets speed improvements up to 50%
Apply OGRFastAtof only on MS runtime libraries where the improvement exist really
I made tests with 'test_ogrsf.exe' and a dbase file. |
@ahuarte47 Looks like it causes a failure in the autotest suite. See Travis log : TEST: ogr_shape_16 ... /home/travis/build.sh: line 509: 18850 Segmentation fault |
@ahuarte47 : yes, that's a bit surprising. the best is that you try locally. Upstream code has never failed on that test. |
Ok, I will try it, I'm new in python and had failed to execute the script by myself |
Hi Even, I have executed the script 'ogr_shape.py' successly. Test Script: ogr_shape ?? |
@ahuarte47 In case you have multiple GDAL installations, are you sure your Python bindings are linking against the libgdal that you are hacking into ? The crash might be specific on Linux too (if you are testing only on Windows) |
Hi Even, you are right, I was executing with python other GDAL installation. I fixed the crash, but there are warning related with OGRFastAtof. I am going review my patch carefully and check if it is really needed. Sorry, I made a bad start in gdal development :-( |
I made more tests with 'test_ogrsf.exe' for some shapefiles. |
Hi Even, you know, had yet to evaluate the loss of accuracy by using 'CPLFastAtof' instead 'CPLAtof' (This patch only applies it on Windows platforms where seems 'atof' is slow). I compared many values from 'atof', 'CPLAtof' and 'CPLFastAtof', and as seen in the modified script ogr_shape.py, accuracy reaches 0.000000001 comparisons. Can you consider to be acceptable? |
Can the fast_atof() function by Tom Van Baak be released under the GDAL X/MIT license ? In your last comparision in the ticket, you mention that "The improvement is more noticeable for large datasets with big linetrings or polygons". But that's a bit fast as a conclusion. In the array, one can see that the improvements linked to ZM are at best 5%. The gain with fast atof seems to be more noticeable, but it is not related to coordinates. atof() is only used for the dbf part, so that should depend on the number of records and the number of real fields. And I don't know if you've investigated what I mentionned before, but for raw rendering (i.e. no conditional rendering based on attributes), QGIS should not trigger any reading of attribute in the shapefile driver if it sets the ignored fields correctly. That would be something to investigate in the first place since it should yield to more performance gain. |
Hi Even, thanks for your reply... As you see, this patch contains two independent improvements. One geometric improvement, and other related to the attribute reading.
The table shows some heterogeneous samples (large geometries, large shapefiles, ...) showing a simple measure of the improvement, not show exact measurements of a specific function, but "perceptions". There layers where user will not perceive improvement but with others the improvement up to 20%, obviously it depends of data. I insist on before, any improvement always will be welcome for the critical task of render a map. For CPLFastAtof, a simple test.... for ( int i = 0; i < 1000000; i++) { double d = CPLFastAtof("1258878.0345"); } It is not unusual to have layers 500.000 records. I know that I have pending study if QGIS configures GDAL to not read the attributes when it is unneeded, and may be the better possible improvement, but in other cases these changes make sense. I also know that there is other very important improvement, at least in Windows, using "filemapping" in a new "virtual file" class as we already spoken. I want implement it with boost library and measure results. do not you think every improvement, however small (~ 5%), not that interesting? Regards |
I'd like to underline that there's always a trade-off between performance improvement and code increased complexity ( GDAL and QGIS could potentially be twice faster if hand-written in optimized assembly, but will you do that ? ) . And that when optimizing stuff, you should always begin with the part that will yield to the most interesting gains. I'd prefer you spending time to study how to avoid qgis to trigger any atof() (being regular, slow or fast) at all when it is not necessary. It will certaily make the fast_atof() stuff unnecessary. And regarding the fast_atof() implementation, it should have a license compatible with the GDAL one. Regarding the geometric changes, it changes shapelib behaviour ( shpopen.c belongs to the upstream shapelib ), so that could potentially necessary a bump in shapelib version when/if they are incorporated into it, or a prominent notice. But that's OK I guess. shapelib activity is mostly due to the OGR shapefile driver I believe. |
Hi Even, thanks for your advice, I then go for the two subjects that a priori seem the best improvements.
you think right? |
Sounds like a good plan |
Hi Even, I have news. As you suggested me I have tested QGIS and I found a bug when are configured the relevant fields of the feature request. Something does not fit, the painting was faster if I changed CPLAtof by CPLFastAtof but this could not be because in single rendering of layers without attributes, fields a priori were not recovered. With your very good advice I have verified the bug and always fecthed all fields. See: Related with my changes in the OGR SHAPE provider, I accept your comments and I am going to implement this feature preserving the current behaviour, I've thought about an optional parameter in other similar 'SHPReadObject' method to manage the internal calloc's of that method. Normal behavior will be as usual, but may pass a new class that overwrite. I find it hard to detail it, so if you think I will implement it in a pull request for evaluation. |
Ok, so if I understand well, the atof optimization is no longer necessary after your fix in QGIS ? Regarding the shapefile reader, you're idea would be to add a SHPReadObject2( SHPHandle psSHP, int hEntity, int bAlwaysAllocZMArrays ) ? The current SHPReadObject() would call SHPReadObject2() with bAlwaysAllocZMArrays = TRUE, whereas shape2ogr.cpp would call it with bAlwaysAllocZMArrays = FALSE. |
Hi Even, I think that atof improvement is always necessary, but now it is a litte less important because of rendering requests only fetch int/double attributes when it is needed. I once studied the code, I would insert this optimization using the SAHooks mechanism when the layer will be open in 'OGRShapeDataSource::OpenFile', it does not change the current behavior of 'SASetupDefaultHooks' as I proposed above. Related with memory allocation I have pulled a new request (#26) with my proposal. It is a bit complex that define a boolean parameter. If you wanted to study the code, I have implemented a heap allocator class, that preserving the current behavior of SHPReadObject, allows to me override it in OGRShapeLayer using a shared memory buffer for all reading requests of SHPObjects. It works because of SHPObject is always a temporal object before create the output OGRFeature object. I think this code improves the heap allocations (Few allocs are executed when a layer is readed) speeding up the data reading of the shape, and it agrees with the current shapelib. I have passed the python autotest suite succesly, and I have measure the improvement with large datasets, but without being exhaustive, getting upto 10-20% of improvement (For compare I only read the shp file). What do you think? |
Finalized. |
This should perhaps help fixing, or at least workarounding, the following issue seen randomly on the gcc52_stdcpp14_sanitize Travis-CI target. e.g on https://api.travis-ci.org/v3/job/351015753/log.txt Running gdrivers/gdalhttp.py... TEST: http_1 ... success ERROR 1: Range downloading not supported by this server! ERROR 1: Request for 8-407 failed TEST: http_2 ... success TEST: http_3 ... success ERROR 4: `/vsicurl/ftp://download.osgeo.org/gdal/data/gtiff/utm.tif' not recognized as a supported file format. TEST: http_4 ... HTTP service for ftp://download.osgeo.org/gdal/data/gtiff/utm.tif is down (HTTP Error: ftp error: 425 Security: Bad IP connecting.) cannot open URL skip TEST: http_5 ... success ERROR 1: JSON parsing error: unexpected character (at offset 0) TEST: http_6 ... success ==31274==WARNING: AddressSanitizer failed to allocate 0x62d00019040f bytes ==31274==AddressSanitizer's allocator is terminating the process instead of returning 0 ==31274==If you don't like this behavior set allocator_may_return_null=1 ==31274==AddressSanitizer CHECK failed: ../../.././libsanitizer/sanitizer_common/sanitizer_allocator.cc:147 "((0)) != (0)" (0x0, 0x0) #0 0x7f3f527259f4 (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x9e9f4) #1 0x7f3f5272a453 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0xa3453) #2 0x7f3f526a7461 (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x20461) #3 0x7f3f527286d5 (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0xa16d5) #4 0x7f3f526acb3d (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x25b3d) #5 0x7f3f526adbd5 (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x26bd5) #6 0x7f3f5271e32e in realloc (/home/travis/build/OSGeo/gdal/install-gcc-5.2.0/lib64/libasan.so.2.0.0+0x9732e) #7 0x7f3f3ea2a940 in VSIRealloc /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsisimple.cpp:814 #8 0x7f3f3e8e3958 in VSICurlHandleWriteFunc /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil_curl.cpp:839 #9 0x7f3f2ff61442 in Curl_client_write (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x16442) #10 0x7f3f2ff8bc46 in Curl_pp_readresp (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x40c46) #11 0x7f3f2ff621ab (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x171ab) #12 0x7f3f2ff64546 (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x19546) #13 0x7f3f2ff61bbf (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x16bbf) #14 0x7f3f2ff61cd1 (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x16cd1) #15 0x7f3f2ff6776b in Curl_disconnect (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x1c76b) #16 0x7f3f2ff7c170 in curl_multi_cleanup (/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4+0x31170) #17 0x7f3f3e902c6b in ClearCache /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil_curl.cpp:2913 #18 0x7f3f3e8fc9f7 in ~VSICurlFilesystemHandler /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil_curl.cpp:2564 #19 0x7f3f3e8fcf23 in ~VSICurlFilesystemHandler /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil_curl.cpp:2569 #20 0x7f3f3e9e4a02 in VSIFileManager::~VSIFileManager() /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil.cpp:1805 #21 0x7f3f3e9e5baa in VSICleanupFileManager /home/travis/build/OSGeo/gdal/gdal/port/cpl_vsil.cpp:1966 #22 0x7f3f3e533e42 in GDALDriverManager::~GDALDriverManager() /home/travis/build/OSGeo/gdal/gdal/gcore/gdaldrivermanager.cpp:262 #23 0x7f3f3e53418d in GDALDriverManager::~GDALDriverManager() /home/travis/build/OSGeo/gdal/gdal/gcore/gdaldrivermanager.cpp:329 #24 0x7f3f3e53a14e in GDALDestroyDriverManager /home/travis/build/OSGeo/gdal/gdal/gcore/gdaldrivermanager.cpp:898 #25 0x7f3f27794ea3 in ffi_call_unix64 (/usr/lib/python2.7/lib-dynload/_ctypes.so+0x1aea3) #26 0x7f3f277948c4 in ffi_call (/usr/lib/python2.7/lib-dynload/_ctypes.so+0x1a8c4) #27 0x7f3f277852c1 in _ctypes_callproc (/usr/lib/python2.7/lib-dynload/_ctypes.so+0xb2c1) #28 0x7f3f27785aa1 (/usr/lib/python2.7/lib-dynload/_ctypes.so+0xbaa1) #29 0x4c2645 in PyObject_Call (/usr/bin/python2.7+0x4c2645) #30 0x537589 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x537589) #31 0x5376f1 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x5376f1) #32 0x5376f1 in PyEval_EvalFrameEx (/usr/bin/python2.7+0x5376f1) #33 0x53e2af in PyEval_EvalCodeEx (/usr/bin/python2.7+0x53e2af) #34 0x536c45 in PyRun_StringFlags (/usr/bin/python2.7+0x536c45) #35 0x53ecb4 in PyRun_SimpleStringFlags (/usr/bin/python2.7+0x53ecb4) #36 0x51e62d in Py_Main (/usr/bin/python2.7+0x51e62d) #37 0x7f3f504657ec in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x217ec) #38 0x41bad0 (/usr/bin/python2.7+0x41bad0) git-svn-id: https://svn.osgeo.org/gdal/trunk@41669 f0d54148-0727-0410-94bb-9a71ac55c965
It is possible improve the shapefile reader avoiding unnecessary memcpy's and calloc's and replacing the CPLAtof by super-fast OGRFastAtof.
OGRFastAtof gets significant improvement at least in OS windows.