-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
renaming/deleting columns takes a lot of time on large vectors #47012
Comments
I'm not totally sure that this is only a DB Manager issue. For example renaming/deleting a column with OGR from a such large vector it takes a fraction of a second.
while is painfully slow in QGIS (for example deleting the same column from the layer properties > fields). There is large margin for improvement here, and as far as I remember this is something that affects QGIS since... forever. |
20 minutes and counting after giving the command to save changes after having deleted a column (within the "fields" tab in layer properties) in the provided sample vector. Ogrinfo > almost instantaneous. |
Wait, to add insult to injury:
|
OSGeo/gdal#5193 will improve thins in particular for renaming that becomes instantenous. |
@rouault thanks! this is on my 10 years old laptop and that 18GB GPKG. Dropping a column is pretty fast! (sorry for the watermark in the cast, but in this period I must use a Windows machine) Recording.4.mp4 |
that's totally unexpected. The whole table should be mostly rewritten (from https://www.sqlite.org/lang_altertable.html "The DROP COLUMN command removes the named column from the table, and rewrites its content to purge the data associated with that column"). Which SQLite version is this: try |
|
@gioman and is the content of the records of the table after column deletion correct ? The PO column should have values at 3333.3321 |
@rouault no! you are right, the PO column got the values of the deleted column... |
ok, that is probably related to what I mentioned in https://trac.osgeo.org/osgeo4w/ticket/726 : "It was available in 3.35.0 but according to https://www.sqlite.org/changes.html corruption could occur until 3.35.5 " |
…5.5, and run foreign_key_check only if foreign_keys is ON (fixes qgis/QGIS#47012)
….35.5, and run foreign_key_check only if foreign_keys is ON (refs qgis/QGIS#47012)
…enaming is to be done and sqlite >= 3.26.0, and do not run integrity_check in that situation (fixes qgis/QGIS#47012)
Alan D. Snow (1): netCDF: always use WKT when found, without comparing with CF params (#4734) (fixes #4725) Alan Snow (2): REF: fix variable name dfMultFactorVerticalShift DOC: Added APPLY_VERTICAL_SHIFT & APPLY_VERTICAL_SHIFT to GDALWarpOptions::papszWarpOptions docstring Alan Thomas (1): DXF: Fix long line handling in edge case Alexander Nehrbass (1): XYZ-driver: Fix incorrect failure to open ASCII-file due to floating point comparison Andrea Giardini (1): Google cloud - Support type=user JSON file for authentication Andrea Giudiceandrea (13): GPKG: fix GDAL_DMD_HELPTOPIC GPKG: add redirection to correct doc page GPKG: fix doc redirection Docs: gdal_rasterize: fix single/double quotes for '-where' parameter (#5241) Add decl_configoption for GDAL_NUM_THREADS in gdal_grid docs Add decl_configoption in jp2openjpeg docs Add decl_configoption in sentinel2 docs Add decl_configoption in XLSX driver docs Add decl_configoption in XLS driver docs Docs: fix a typo in gdal_calc Docs and help strings: add missing -pct option description in pct2rgb and fix some typos in pct2rgb and rgb2pct (#5447) Fix gdal_grid link in XYZ driver docs Fix gdal_edit link in ECW driver docs Andreas Degwerth (Airbus) (1): NITF: RPF.toc Skip plausibility check for Overviews and Legends; disable some checks for polar zones (#5654) Andrew C Aitchison (2): CMake: make OGDI driver build on Ubuntu 21.10 (#4905) (fixes #4817) OGREnvelope: ignore fp warnings for operator== like IsInit() Antonio Valentino (1): VRT: add div, polar, exp pixel functions. improvements in sum, mul, inv (#5308) Bas Couwenberg (1): Update BAG 2.0 Extract URLs Björn Harrtell (1): FlatGeobuf: If CRS WKT detected to be non UTF-8, force it to ASCII (#5062) Brendan Jurd (1): PG: skip all leading whitespace in SQL statements. (#4787) C41eb (1): Python bindings: Change GetFieldAsBinary to use VSIMalloc. (#4774) Chiara Marmo (1): Migrate doc building to github action workflows (#5182) Chris Barker (1): Updated install instructions Chris Cantey (1): Update python.rst (#5658) Cédric Traizet (1): Fix broken link for the news page Daniel Baston (1): gdalwarp: Test and document mode behavior in case of ties Dominik Michael Rauh (1): port/cpl_recode_iconv.cpp: fix invalid cast error with uclibc (#5684) Dominik Nussbaumer (1): rdb: enable building rdb driver via CMake (#4816) Egor Pugin (1): Initialize jxl structures with zeros. Fixes VS17.0 build error. Closes #4842. Even Rouault (1202): .gitignore: remove CMakeLists.txt and .cmake files autotest/CMakeLists.txt: fix for cmake < 3.14 charls: use 'CharLS::charls' case test_cpl.cpp: avoid redefinition of GDAL_COMPILATION CMake: fix CONFIG_DATA to not include GDAL version port/CMakeLists.txt: restrict exported headers to those of port/GNUmakefile CMake: install headers in prefix/include, for compat with current practice CMakeLists.txt: remove explicit cmake_policy() that are enforced by the 3.10 min requirement, and add comments for remaining ones CMake: make us resilient to absence of autotest directory, as found on tarballs CMake: detect issue with conflicting cpl_config.h CMake: only set POSITION_INDEPENDENT_CODE ON for BUILD_SHARED_LIBS=ON frmts/CMakeLists.txt: make more formats optional CMake: change cpl to an object library, to help with static builds CMake: make bindings build optional update_copyright.py: move under scripts/ port/CMakeLists.txt: simplify unity build setting, and better linking properties for cpl library test_gdal.cpp: do not crash if PCIDSK driver not available mrf.py: fix when driver not available cmake/template/pytest.ini.in: add gdrivers/data to -norecursedirs frmts/CMakeLists.txt: JPEG is required for TIFF driver cmake: make gdal_dependent_format/ogr_dependent_driver default to OFF if GDAL_BUILD_OPTIONAL_DRIVERS/OGR_BUILD_OPTIONAL_DRIVERS is OFF CMake: make a few more formats optional gcore/CMakeLists.txt: fix OGR_ENABLE_SQLITE=OFF builds CMake: fix MRF build CMake: remove possibility to build against external PCIDSK SDK, as it is not commonly found .cmake-format.py: import from https://github.com/miurahr/cmake4gdal CMake: apply cmake-format -i CI: add .github/workflows/cmake-lint.yml (adapted from https://github.com/miurahr/cmake4gdal) alg/: does not depend on SFCGAL CMake: add explicit linking to gmpxx for SFCGAL 1.3.7 CMake: fix paths to now obsolete gdal/ subdirectory CMake: fix gdal-config --libs generation for the 'dl' library added through CMake: move inclusion of libcsf subdirectory to frmts/pcraster/CMakeLists.txt gdal.cmake: tune how we link gdal to GDAL_LINK_LIBRARY CMake: only apply -msse/-mavx to gdalgridsse.cpp/gdalgridavx.cpp CMake: fix internal libjpeg build CMake: fix CAD building on Windows CMake: default to GDAL_HIDE_INTERNAL_SYMBOLS=ON CMake: remove -DBIGTIFF_SUPPORT CMake: fix URL to libtiff project CMake: simplify a bit ZSTD TIFF conf CMake: autotest: fix copying of autotest content on Windows CMake: fixes related to internal libtiff/libgeotiff renaming (enable it by default) Remove obsolete tests and uses of dbmalloc.h Remove STDC_HEADERS setting / checks CMake: improve logic to generate gdal_version.h, and remove the sha1sum from the GDAL_VERSION CMake: add trick w.r.t gdal_version.h to minimize the amount of rebuild CMake: fix build issue with ninja and Python bindings CMake: remove dbmalloc / stdc_headers related stuff with MSVC CMake: remove unused TARGET_LINK_LIB CMake: fix linking to internal libjson in ogr component CMake: turn all STATIC libs to OBJECT libs, and add their link libraries through a global property cmake: fix gdal-config CONFIG_DEP_LIBS value for -lm CMake: fix FindPoppler.cmake for Windows / conda CMake: fix include order for GTiff driver CMake: make alg depend on generate_gdal_version_h CMake: generate_gdal_version_h.cmake: avoid error when git not found .github/workflows/cmake_builds.yml: disable all jobs but build-linux-ubuntu-focal CMake: fix build of GPKG driver CMake: fix build of autotest/cpp on mingw64 .github/workflows/cmake_builds.yml: enable build-windows-conda CMake: autotest/cpp: do not modify PATH as it doesn't work with MSVC CMake: only run test-copy-words on build-windows-conda config cmake: FindNetCDF: fix retrieval of NETCDF_HAS_xxxx features CMake: rik: fix inclusion of Zlib CMake: internal libjson: set _XOPEN_SOURCE to 700 CMake: fix shapelib build regarding SAOffset CMake: geoconcept: add MSVC warning suppression CMake: internal libtiff: tune include order CMake: avoid incompatible configuration regarding internal libtiff/libgeotiff CMake: emit warning about experimental status CMake: add extra warnings from configure/nmake CMake: add -Werror to build-linux-ubuntu-focal job CMake: fix NETCDF_HAS_NC4 detection on Windows CMake: fix use of wrong variable in GdalVersion.cmake CMake: add missing dependencies to generate_gdal_version_h CMake: replace CMAKE_[SOURCE/BINARY]_DIR witch PROJECT_[SOURCE/BINARY]_DIR to be able to add_subdirectory(gdal) CMake: test building as a add_subdirectory() CMake: fix LIBKML finding CMake: disable libkml on build-windows-conda CMake: add JPEG12 support CMake: add gdal_create and gdal_viewshed cmake/modules/DefineFindPackage2.cmake: fix typo cmake/modules/DefineFindPackage2.cmake: other typo fixes CMake: remove BPG detection CMake: CheckDependentLibraries.cmake: typo fix CMake: slightly clearer logic regarding rasterlite2 CMake: avoid rebuild of tif_jpeg* files Add scripts/reformat.sh: to reformat CMakeLists.txt files for now CMake: remove trace of BPG driver CMake: fix HINSTS typo, and also fix Rasterlite2 package config variable names CMake: rename ENABLE_LTO as ENABLE_IPO CMake: idrisi: remove unappropriate plugin test CMake: remove USE_CPL option CMake: make it possible to disable a lot of found dependencies CMake: Doc: initiate a documentation page (WIP !) CMake: revise how builtin vs plugin drivers are declared CMake: fixes regarding plugin build CMake: fix minimal build CMake: enable MITAB driver CMake: adjust how to specify a driver name for option names when it is not the same as the subdirectory CMake: fix JPEG2000/Jasper driver CMake: GTiff: fix minimal build CMake: pytest.ini.in: add deprecation driver suppressions CMake: make OpenCL opt-out, and disable it by default CMake: add libdeflate support CMake: add LZ4 support CMake: FindLZ4.cmake: although look for liblz4 (for Windows Conda) CMake: FindLibLZMA: remove unneeded extra function checks (for Windows Conda) CMake: gtiff/mrf: modify to have internal library includes first CMake: add blosc support CMake: fix getting LibLZMA,INTERFACE_INCLUDE_DIRECTORIES CMake: add TileDB support CMake: FindTileDB.cmake: fix warning with cmake >= 3.19 CMake: FindTileDB.cmake: remove useless variable CMake: improve doc CMake: doc: use 'compilers' package CMake: use internal shapelib by default CMake: remove detection of SDE, since the drivers have already been removed CMake: make CharLS package disableable CMake: make CFITSIO disableable CMake: make Armadillo disableable CMake: alg/CMakeLists.txt: remove useless target_compile_definitions CMake: changes to have the same set of headers files installed as for autoconf builds CMake: reorder inclusions so that internal libraries are included first CMake: cleanup and fix setting INST_DATA define to find resource files CMake: fix issues spotted by cmake --warn-uninitialized CMake: fix Windows build CMake: remove FindTileDB.cmake that is not needed CMake: define CPL_DISABLE_DLL for static MSVC builds CMake: remove unexisting cmake 3.9 directory CMake: revise version numbers in libgdal.so* files to match libtool convention CMake: autotest/CMakeLists.txt: correctly use CMAKE_HOST_SYSTEM_NAME CMake: suppress warnings for Windows builds of internal libjson CMake: add warnings suppression for MSVC on internal libz and libtiff CMake: add warnings suppression for MSVC on internal libjpeg CMake: make MSVC debug postfix customizable CMake: number of fixes and improvements for Python bindings CMake: PCIDSK: enable JPEG support and fix build as plugin CMake: PCRaster: enable build as plugin CMake: fix -fvisibility=hidden CMake: CAD: fix build as plugin CMake: make PNG driver PLUGIN_CAPABLE CMake: make CALS driver PLUGIN_CAPABLE CMake: netCDF: fix MEM and UFFD support CMake: Cloudant driver can't be plugin due to dep to CouchDB driver CMake: LVBAG: add GEOS dependency CMake: pass SQLITE_HAS_COLUMN_METADATA to SQLite and GPKG drivers CMake: make WCS and HTTP driver separately buildable CMake: make WMS driver plugin capable CMake: fix building EXR driver as plugin CMake: make Java bindings so/dll link against libgdal CMake: add -Wl,--no-undefined to detect issues with drivers built as plugins CMake: fix build against Poppler 2021.08 CMake: disable bindings when doing a static build CMake: robustify use of -Wl,--no-undefined which doesn't work on Mac CMake: doc: fix CMAKE_INSTALL_PREFIX vs CMAKE_PREFIX_PATH CMake: make name of 'gdal' target configurable CMake: change name of library target to GDAL upper case, so that GDAL::GDAL can be imported conforming to existing FindGDAL.cmake distributed with cmake CMake: make GDAL usable as add_subdirectory() of another project CMake: FindFreeXL.cmake: look for freexl_i first, for Conda shared library CMake: make DAP disableable CMake: tune find and use of PDFium CMake: make use of libxml2 optional CMake: make Jasper disableable CMake: CI: reenable msys and mac builds CMake: make MySQL disableable CMake: CI: disable mysql for msys2-mingw CI: workaround a regression / change of behaviour in JXL CMake: CI: disable python bindings for msys2-mingw typo fixes port/CMakeLists.txt: use correct BUILD_SHARED_LIBS name scripts/fix_typos.sh: adapt for tree reorganization typo fixes CMake: for Windows multi-generator, only generate Python bindings in Release mode CMake: cleanup setup.py.in VICAR: fix harmless unsigned-integer-overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40377 VFK/Selafin: avoid issues with GPKG driver when those drivers are built as plugins apps/GNUmakefile: generate test_ogrsf by default (but not installed) Export VSI_TIFFOpen() to build the INGR driver as plugin CMake: make sure that gcore/ if built with OGR_ENABLE_SQLITE Doc: fix doc building gcore/CMakeLists.txt: make sure OGR_ENABLE_SQLITE is defined CMake: gcore/CMakeLists.txt: use correct variable for Xerces CMake: python bindings: use parallel build README.md: update links [ci skip] doc/.azure-pipelines.yml: fix path CMake: update resource file list to install CMake: fixes for gdal.pc and gdal-config generation GML/NAS: avoid false positive -Wnull-dereference -O2 warnings Warp kernel: avoid writing outside allocated buffer if more threads than needed are allocated warp VRT: detect inconsistent block size between dataset and bands (fixes #4714) warp VRT: do not serialize block size at band level, since already serialized at dataset level (refs #4714) Warped VRT: advertize INTERLEAVE=PIXEL for faster processing of multiband VRTWarpedDataset::ProcessBlock(): fix issue in the unlikely situation where a block would be > 4 GB GDALAutoCreateWarpedVRTEx(): avoid potential crash in case of error autotest: typo fix gdal_pansharpen.py: display usage if not enough filenames provided BMP: harden identify checks to avoid misidentification of other datasets (fixes #4713) CMake: Python: do not regnerate docs/.i file to avoid issues with make clean CMake: Python: remove obsolete tests on numpy version CMake: tweaks so that python 3.6 is detected out of the box on ubuntu 18.04 CMake: detect and use -Wzero-as-null-pointer-constant CMake: fix case of tiledb target, so it works properly as plugin HOWTO-RELEASE: fixes related to new tree organization [ci skip] CMake: FindOpenJPEG: add include directories for newer versions CMake: fix running unit tests under Windows CMake: set HAVE_STD_IS_NAN in cpl_config.h autotest/cpp/testclosedondestroydm.cpp: do not error out on JPEG2000 driver Shorten DO_NOT_DEFINE_GDAL_RELEASE_DATE_AND_GDAL_RELEASE_NAME macro name to avoid MSVC warning scripts/setdevenv.sh: adapt it for CMake builds CONTRIBUTING.md: updates for new tree org + CMake hints swig/python/gdal-utils/scripts: make scripts executable CMake: fix to avoid cmake to be reinvoked everytime gcore/gdalsse_priv.h: fix warning about cast CMake: extent/fix autotest/cpp tests Doc: netcdf: fix default value for GDAL_NETCDF_IGNORE_XY_AXIS_NAME_CHECKS (refs #4732) CMake: fix MSVC build HDF5: fix issue when netCDF and/or HDF5 drivers built as plugins with multidim datasets fuzzers/: make gdal_fuzzer and ogr_fuzzer build on Windows CMake: add ctest'ing fuzzers/build.sh: adapt path for new tree org [ci skip] CMake: do not include some SSE/AVX tests in ctest list CMake: fix AVX detection on Windows and other SSE/AVX fixes Doc: advertize 3.3.3 release CMake: add -relativeimport when regenerating Python bindings Solaris compilation fix (master only) CMake: formatting fix CMake: fix SQLITE_HAS_COLUMN_METADATA detection, and use of PCRE lib CMake: remove unneeded target_include_directories() that are got through gdal_target_link_libraries() CMake: add missing dependency of ogrsf_frmts to generate_gdal_version_h m4/acinclude.m4: fix detection of fseeko/ftello on netBSD port/cpl_conv.cpp: fix compilation problem on netBSD CMake: fix setting of ICONV_CPP_CONST in cpl_config.h test_osr_ct.cpp: fix wrong use of abs() (spotted on netBSD) gdal_unit_test: fix failure spotted on netBSD on test_cpl<4> CMake: check that SQLite3 has the RTree extension CMake: rename SQLite_HAS_xxx to SQLite3_HAS_xxx for consistency netCDF: limit SetFromUserInput() use to non file input CMake: fix dependency to Blosc in Zarr driver Move tif_float.c/h from GTiff driver to port/, to be able to build Zarr driver as plugin Export symbols for Zarr driver as plugin CMake: allow Zarr driver to be built as plugin CMake: build as many drivers as possible as plugins in Windows Conda build CMake: fix autotest targets with Ubuntu 18.04 CMake: fix i386 builds autotest/cpp: do not use assert() CMake: exclude testepsg from default build CMake: fix test-fuzzer-* on Windows TileDB: fix crash when creating array from subdatasets fails scripts/cppcheck.sh: update for cppcheck 2.7 contour.cpp: remove unneeded nullptr check TileDB: fix handling of relative paths on Windows Embeded Python: fixes to load Conda and mingw64 python on Windows Embeded python: list Python 3.10 netcdf.py: Windows fix CPLRecode(): fix recoding between UTF-8 and CP_ACP/CP_OEMCP on Windows build that have iconv support netcdf_multidim.py: Windows fix netCDF: disable filename recoding to ANSI on Windows for netCDF >= 4.8 CMake: do not build HTTP driver as plugin pds4.py: make it Windows robust ogr_gmlas.py: Windows fix Python bindings: for Windows and Python >= 3.8, automatically add path with [lib]gdal*.dll with os.add_dll_directory() CMake: reformat TileDB: fix memory leak (introduced in recent commit) test_gdalsrsinfo.py: fix when CPL_ERROR_SEPARATOR=; is set [ci skip] tiff_write.py: adapt checksum for Windows Conda builds [ci skip] CMake: autotest: fix use of bad variable Rename NEWS to NEWS.md [ci skip] mkgdaldist.sh: remove autom4te.cache dir from archive (fixes #4748) [ci skip] GDALmake.opt.in: update LIBGDAL_CURRENT to the one of the 3.4 branch CMake: CI: enable all ctest for Linux and Windows conda builds CMake: fix driver enabling conditions CMake: JPEG must be defined before GTiff CMake: Windows CI: set git core.autocrlf to false CMake: set LIBDEFLATE_SUPPORT for internal libtiff when appropriate GTiff: explicitly enable strip choping test_gdal_viewshed.py: tolerate more error messages (needed for Conda cmake builds) CMake: CI: skip test_ogr_gmlas_huge_processing_time() on Windows Conda builds MITAB: allow reading MID/MIF files with lines up to 1 million bytes (fixes #3943) cpl_virtualmem.cpp: add comment (fixes #811) [ci skip] ogr_carto.py: skip test_ogrsf test on CI (fixes CMake Conda windows failures) OGRSpatialReference::GetName(): workaround a PROJ 8.2.0 bug for BoundCRS gdaladdo -clean: remove overviews of mask (fixes #1047) ODS: avoid huge memory allocations on files abusing repeated cells. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40568 CMake: improve install of Python bindings, and add doc CMake: set SKIP_BUILD_RPATH YES property on plugins. Helps for Linux builds CMake: remove useless target properties on binaries CMake: fixes for MacOSX framework builds DGNv8: fix build issue due to FlushCache() changes CMake: set rpath to installed plugins on MacOSX frameworks (fixes #4761) CMake: add support for building the MSG driver GTiff: JXL codec: use non-deprecated method to set encoding effort Unix builds: move generated headers to a generated_headers subdirectory configure.ac: fix detection of OpenEXR >= 3 (fixes #4766) CMake: detect OpenEXR >= 3 CMake: add JXL support CMake: add support for SDK 5.5 ECW: fix test failures with ECW 5.5 Doc: precision for ECW [ci skip] CMake: fix MrSID support Fix OSSFuzz build [ci skip] CMake: fix KEA support HDF5 and KEA: remove warnings with MSVC Disable/fix various compiler warnings on CMake Conda Windows builds MSG driver: fix/workaround MSVC warnings CMake: MSG: set -DWIN32 CI: turn warning-as-errors on CMake Conda Windows builds CMake: fix warnings on Mac builds and turn on warning-as-errors Doc: advertize 3.4.0 CI: disable building C# samples on Windows due to #4792 Doc: download.rst: fix link FileGDB: add a WKID (undocumented for now) layer creation option GDALDriver::QuietDelete(): take into account papszAllowedDrivers argument GDALDataset::MarkSuppressOnClose(): delete auxiliary files (fixes #4791) CMake: fix typo in FindSQLite3.cmake (fixes #4795) CMake: FindECW.cmake: add comment (fixes #4796) Doc: fix links to no longer existing gdal/ subdirectory (fixes #4798) GTiff: only emit warnings on libgeotiff PROJ errors (fixes #4801) gdalinfo --build: report PROJ version of build and runtime Doc: better point to RPC transformer options CMake: fix detection of SQLite3 extra symbols when linking with a SQLite3 static lib (fixes #4811) gdal_polygonize.py: remove use of Unicode double quote characters Docker: fix --release builds CMake: implement Kakadu library finding Zarr: be robust to duplicated array and group names in NCZarr metadata, which could lead to performance issues. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40949 CMake: SQLite3: accept variables in upper case ogrsqlite3ext.h: avoid 'redefinition of typedef 'sqlite3_api_routines' is a C11 feature' warning CMake: force C99 FileGDB: avoid warning with MSVC CMake: add FileGDB support Add support for PCRE2 (to replace deprecated PCRE) (fixes #4822) CMake: add PCRE2 support (refs #4822) OGRSimpleCurve::segmentize(): avoid too many memory reallocation (refs #4826) gdalwarp -crop_to_cutline: relax epsilon to avoid extremely long loop (fixes #4826) CMake: add support for LURATECH GRIB: fix thread-safetey of errSprintf() (#4830) Oracle: fix CMake support and fix compiler warnings (#4812) /vsis3/: allow setting AWS_PROFILE to a profile that uses IAM role assumption CMake: fix build issue when re-runing cmake with PCRE present but missing PCRE2 (fixes #4845) GPKG: fix nullptr dereference on corrupted databases with sqlite >= 3.35 CMake: link wsetargv.obj to GDAL programs with MSVC (fixes #4855) CI conda: fix path to GDAL sources Revert "Github workflows: disable Conda builds in pull requests (refs #4599)" github/workflows/conda.yml: force macos-10.15 to maybe fix build issue CPLRecodeIconv(): avoid potential unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41201 COG: only create RGB JPEG with mask if 4-band is alpha (fixes #4853) GTiff: avoid warning with >= 5 band and JPEG compression GDALPamRasterBand::CloneInfo(): deal correctly with NaN nodata to avoid generating useless .aux.xml file (fixes #4847) GRIB: avoid read heap buffer overflow due to inappropriate split-and-swap on dataset with weird georeferencing. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41260 (3.4.0 regression) CMake: for default Python_FIND_STRATEGY=LOCATION, requires Python 3.6 at least (avoid picking Python 2.7 on Mac) CMake: avoid warning messages for rdb and TileDB CMake: FindKDU.cmake: fix for CMake < 3.13 ogrsqliteregexp.cpp: fix control flow issue in PCRE2 code. Fixes CID 1466404 ZIP support: avoid warnings on MacOS SQLite: workaround MacOS system SQLite non-default settings that cause issues with WAL and AlterFieldDefn() when patching CREATE TABLE DDL VFK: avoid crash when a SQLite3 statement fails ogr_virtualogr.py: skip when SQLite built with OMIT_LOAD_EXTENSION ogr_vdv.py: workaround test failure with MacOS system SQLite ogr_vfk.py: workaround test failure with MacOS system SQLite CMake: avoid issues with Mono framework on MacOSX CMake CI: enable tests on Mac builds TileDB: avoid warnings about deprecated functions TileDB: avoid clang warning about parenthesis cpl_compressor.cpp: avoid warning when including lzma.h of Conda with CLang cpl_xml_validate.cpp: avoid warning with libxml2 of Conda with CLang port: avoid warnings with zlib of Conda with CLang SWIG: add missing CPLES_SQLI constant (fixes #4878) FindGeoTIFF.cmake: locate release and debug libraries (fixes #4854) FindPROJ.cmake: locate release and debug libraries (fixes #4854) OpenFileGDB: correctly parse raster fields of type == 2 which are inlined binary content (fixes #4881) GTIFFWriteDirectory(): remove useless paramater External overviews: automatically turn PLANARCONFIG_CONTIG for WebP overviews GTiff: LERC overview related improvements OGR_CT: add a OGR_CT_PREFER_OFFICIAL_SRS_DEF config option /vsiaz/: read credentials from ~/.azure/config as an additional fallback method gdal_sieve.py: fix exception when source dataset has no nodata value (3.4.0 regression) (fixes #4899) CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe ossfuzz build: build GDAL against external libjpeg since newer poppler requires jpeg_mem_src() which is new to libjpeg v8 and thus absent from our internal libjpeg v6b ElasticSearch: do not try to open reserved .geoip_databases to avoid error message with recent Elastic versions Internal libtiff: resync with upstream Internal libgeotiff: resync with upstream Shape: better deal with ETRS89 based CRS with TOWGS84[0,0,0,0,0,0,0] GDALDatasetCopyWholeRaster(): clarify INTERLEAVE option meaning and raises warning if invalid value is specified (fixes #4909) WFS: if COUNT is present in WFS >= 2, use it as the page size. Also clarify doc SQLite: add a STRICT=YES layer creation option to create tables as SQLite >= 3.37 STRICT tables PDS4: write conformant Equirectangular when input raster is a geographic CRS Zarr: hopefully fixes 'runtime error: reference binding to null pointer of type 'unsigned char''. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41517 Zarr: fix cppcheck warning added in previous commit ODS: avoid crashing 'floating-point exception' when evaluating -2147483648 % -1. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41541 GTiff: fix performance issue when reading transfer functions (fixes #4923) Internal libtiff: resync with upstream /vsiaz/: allow authorization through an access token specified with the AZURE_STORAGE_ACCESS_TOKEN config option (to be used with AZURE_STORAGE_ACCOUNT) Move multi-line CSV parsing function from OGR driver to cpl_csv.h MITAB: fix parsing .mid files with newline character in string field (fixes #4789) GRIB: avoid read heap buffer overflow due to inappropriate split-and-swap on dataset with weird georeferencing. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41637 (3.4.0 regression) GTiff: propage SPARSE_OK to overviews (refs #4932) Fix -Wnull-dereference warnings with gcc in -O3 -DNDEBUG GTiff: avoid huge memory allocation when generating overviews on large single-band 1-bit tiled files (fixes #4932) SQLite: fix memory leak recently introduced (master only). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41671 tif_jxl.c: adapt for latest jxl master API changes .github/workflows/ubuntu_20.04/build-deps.sh: force rebuild of dep Revert "CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe" TopoJSON: fix duplicate 'id' field, and other potential issues when reading fiels (3.4.0 regression) Internal libtiff: fix issue with rewrite-in-place logic of https://gitlab.com/libtiff/libtiff/-/issues/309 PCIDSK: fix write heap-buffer-overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41993 frmts/georaster/CMakeLists.txt: use correct variable to decide to link against external libz configure: change default of RENAME_INTERNAL_LIBTIFF/LIBGEOTIFF/SHAPELIB_SYMBOLS to yes Unix/CMake build: when building against internal libjpeg, prefix by default libjpeg symbols with gdal_ (fixes #4948) CMake: add support for Teigha/ODA libraries Internal libtiff: resync with upstream to get https://gitlab.com/libtiff/libtiff/-/merge_requests/277 GDALDefaultCSVFilename(): remove remains of GeoTIFF EPSG CSV files era Typo fixes GRIB: fix writing negative longitude of natural origin for Transverse Mercator, and fix reading it for TMerc, LCC, ACEA and LAEA Internal libtiff: LZW codec: fix support for strips/tiles > 2 GB on Windows tif_lzw.c: fix warnings of previous commit tif_lzw.c: other warning fixes PG: error out if non UTF-8 content is transmitted in COPY mode, when client_encoding=UTF8 (refs #4751) S57: enable recoding to UTF-8 by default (RECODE_BY_DSSI=YES open option) (fixes #4751) tiff_write.py: typo fix [ci skip] COG: fix potential generation failure when main imagery has overview and mask none Add GDALPamDataset::DeleteGeoTransform() (refs #4877) GTiff: make SetGCPs(), SetGeoTransform(), SetSpatialRef(), SetNoDataValue(), SetMetadata[Item]() write to PAM .aux.xml on read only files (fixes #4877) test_gdal_edit.py: test -ro switch (refs #4877) Doc: fix broken links to /vsi file systems (fixes #4967) Internal libtiff: have TIFFJPEGIsFullStripRequired properly renamed for RENAME_INTERNAL_LIBTIFF_SYMBOLS builds and simplify tif_jpeg build frmts/gtiff/libtiff/CMakeLists.txt: adapt from previous commit fuzzers/build.sh: fix PROJ linking GTiff: add support for reading/writing color table from/into PAM .aux.xml (fixes #4897) PCIDSK SDK: fix harmless unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42120 Remove obscure test program xmlreformat (that was not built by default) nmake.opt: fix wrong variable name in example MSODBCSQL_LIB [ci skip] CMake: fix ODBC on Windows, and detect MSSQL native CLI or ODBC (fixes #4810) GTiff: do not warn about buggy Sentinel1 geotiff files use a wrong 4326 code for the ellipsoid CMake: fix internal libtiff build with jpeg12 CMake: fix MSVC Ninja build FindMSSQL_ODBC.cmake: add Linux support CMake CI Linux: add msodbcsql17 client support MSSQL: fix warnings on Linux MSSQL: avoid GetLayerDefn() to return null, and call it in GetExtent() OGRMSSQLSpatialTableLayer::GetLayerDefn(): fix potential memory leak of pszFIDColumn OGRMSSQLSpatialDataSource::DeleteLayer(): remove suspicious test 'strlen(pszTableName) == 0' that causes a read after free MSSQL: fix issue when inserting strings in bulk copy mode on Linux MSSQL: make GetNextFeature() end bulk copy mode (to fix autotest failure) MSSQL: CreateFeature in bulk copy mode: make it use feature geometry SRID when set (to fix autotest failure) MSSQL: disable bulk mode when a UUID field is found, as this doesn't work currently ogr_mssqlspatial.py: adapt test for bulk copy mode MSSQL: do not set field width when reading smallint/int/bigint/float/real columns, and correctly roundtrip smallint/Int16 (fixes #3345) Metadata readers: remove thread-unsafe use of localtime() Various cppcheck fixes (mostly always true/false conditions) cppcheck fix: avoid 'Returning pointer to local variable 'poInvisibleLayer' that will be invalid when returning' CMake: fix cryptopp support, and some tidying in check_cxx_source_compiles() of other packages CMake: fix GTA driver CMake: disable OpenMP detection CMake: fix Crnlib detection CMake: remove detection of unused epsilon package CMake: add support for IDB fuzzers/gdal_fuzzer.cpp: avoid int64 overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42251 FindIDB.cmake: add advanced options Revert "Revert "CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe"" CMake: fix FYBA/SOSI support CMake: add support for MongoDBv3 driver CMake: fix support for RasterLite2 and Spatialite CI CMake Ubuntu: add a few missing dependencies CMake: add support for HEIF CMake: add GDAL_ENABLE_FRMT_JP2MRSID option CMake: mark number of options as advanced CMake: use generic infrastructure for GTA and KEA detection CMake: use generic infrastructure for PDFium detection CMake: use generic infrastructure for CFITSIO and JXL detection add_javadoc.c: robustness fixes CMake: remove duplicated FindRasterlite2.cmake swig/java/add_javadoc.c: fix bad use of strcat() that caused crashes on MacOSX CMake: fix Java bindings CMake Java: avoid constant regeneration of maven .jar when GDAL_JAVA_GENERATE_JAVADOC=OFF Internal libtiff: resync with upstream [ci skip] OGR GMT: allow writing to /vsistdout/ (fixes #4993) JP2KAK: add support for reading/writing Int32/UInt32 data types ECW: do not try to open UInt32 JPEG2000 if the SDK is buggy (fixes #3860) CMake Python: small tuning to avoid useless run of setup.py PCIDSK SDK: fix harmless unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42420 gdal_polygonize.py: make -8 switch work again (fixes #5000) CMake: add/fix CSharp support CMake CSharp: remove .config files for Unix CMake CSharp: add install target test_gdalinfo.py / test_gdalinfo_py.py: avoid potential conflicts if run in parallel Warped VRT: fix issue with blocks without sources and alpha band (fixes #4997) Remove RDA (DigitalGlobe Raster Data Access) driver gdalwarp: do not emit "Point outside of projection domain" / "tolerance condition error" (fixes #4934) CMake DWG: build createdgnv8testfile utility DWG/DGNv8: avoid potential crash with ODA 2022 on Linux CMake ODA: add a TEIGHA_ACTIVATION_FILE_DIRECTORY variable Remove most uses of GDALIsInGlobalDestructor() CMake: fix CSharp build on Windows Internal libtiff: resync CMake FindTEIGHA.cmake: update for ODA >= 2021 on Windows CMake: fix Visual Studio ALL_BUILD solution swig/python/GNUmakefile: fix/workaround setuptools 60.0 and Debian --install-layout CMake Python: fix/workaround setuptools 60.0 and Debian --install-layout generate_quant_table_md5sum.cpp: use uint16_t (fixes #5021) configure: detect xlocale.h to use LC_NUMERIC_MASK on Mac (fixes #5022) cmake: detect xlocale.h to use LC_NUMERIC_MASK on Mac (fixes #5022) USGSDEM: fix reading datasets with 1025 byte records ending with linefeed (fixes #5007) autoconf/nmake: Add support for external libqhull_r (refs #4040) CMake: add support for external libqhull_r (refs #4040) gdal2tiles: XML escape input filename (fixes #5032) CMake: fix build with cmake < 3.12 CMake: cleanup w.r.t internal libqhull delaunay.c: fix warning with MSVC with libqhull_r Internal libqhull: update to qhull_r 2020.2 (8.0.2) (fixes #4040) internal_libqhull/poly_r.c: avoid int overflow on 32bit (re-apply 4224afd669eebbf657afe90014436bdc83fd0f34) Drop support for non-reentrant external libqhull GDALTriangulationCreateDelaunay(): allow redirecting error stream of GDALTriangulationCreateDelaunay() to temporary file to avoid issues with ctest with MSVC and external libqhull CMake Windows Conda: use external qhull CMake: reformat gdal.cmake and CheckDependentLibraries.cmake CMake: make Armadillo link to LAPACK if needed (Windows Conda builds) openexr_headers.h: avoid warnings with MSVC CMake: fix build on Linux with Conda, related to iconv Avoid warnings related to libxml2 Avoid warnings related to curl Avoid warnings related to libgeotiff CPLZLibInflate(): workaround issue with /opt/intel/oneapi/intelpython/latest/lib/libz.so.1 from intel/oneapi-basekit Docker image CMake: disable -ffinite-math-only with Intel's icx compiler Internal libqhull: silence Coverity Scan warnings Internal libtiff: resync with upstream static'ify method Remove redundant nullptr checks CITATION: update year Tag another method as static cppcheck.sh: use --library=gnu to avoid warning with anonymous mmap() and fd==-1 autotest/gcore/pixfun.py: attempt at fixing failure on CI build-linux-ubuntu-focal gdal2tiles: remove/fix broken links Docker: update Alpine based images to 3.15 [ci skip] internal libqhull: fix / silence Coverity warnings gdal2tiles: lower chuncksize value to allow parallel processing when generating ~ 100 tiles docker/ubuntu-full/Dockerfile: fix for building libjxl [ci skip] Doc: advertize 3.4.1 release metaname.cpp: white space fixes degrib: partial resynchroinzation with degrib 2.25 (degrib-20200921) degrib: fix cppcheck warning cpl_csv.h: add a CSVRewind() function GRIB: mode degrib hard-coded tables to .csv files in the data/ directory GRIB: merge content from WMO tables at https://github.com/wmo-im/GRIB2 with DEGRIB ones GRIB: fetch WMO 4.2 subtables Elasticsearch: add a AGGREGATION open option (#4962) configure.ac: add a --with-qhull-pkgname=qhull_r/qhullstatic_r option to select with qhull package to use CMake: add a QHULL_PACKAGE_NAME=qhull_r/qhullstatic_r option to select with qhull package to use GRIB: use tagged version of the wmo-im/GRIB2 repository Doc: fix warnings with Doxygen 1.9.3 configure.ac: remove useless QHULL_INCLUDE_SUBDIR_IS_LIBQHULL ogr2ogr: propagate error in final CommitTransaction() to status code of the utility (fixes #5054) PG: propagate errors in defered EndCopy() call in CommitTransaction() (fixes #5054) Elastic: support GetLayerByName() with multiple layers and/or wildcard (fixes #4903) HDF5: support libhdf5 1.13.0 (fixes #5061) frmts/pdf/GNUmakefile: force c++17 with Poppler > 21 (fixes #5071) frmts/pdf/CMakeLists.txt: force c++17 with Poppler > 21 (fixes #5071) ogr2ogr / PG driver: make ogr2ogr -lco GEOM_TYPE=geography work when the source layer has a named geometry column, and also reject creating geography columns with a SRS != EPSG:4326 (fixes #5069) GPKG/SQLite: deal explicitly with CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES for /vsi network file systems (fixes #5031) Python bindings: replace unsafe use of tempfile.mktemp() by mkstemp() (fixes https://huntr.dev/bounties/59e93871-266a-40be-9aad-4576d3dc48f2/) PG: support other geographic SRS than EPSG:4326 for geographic type (fixes #5075) gdal2tiles: fix issue with multiprocessing and the gdal2tiles launcher script on Windows and Python >= 3.8 (fixes #4951) autotest/pymod/test_py_scripts.py: run wrapper scripts of swig/python/gdal-utils/scripts instead of swig/python/gdal-utils/osgeo_utils GRIB: avoid harmless unsigned integer overflow. master only. fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43479 docker/README.md: more details about grids [ci skip] doc/source/build_hints.rst: fix copy&paste error CMake python: move .in files to swig/python CMake: Python: fix setup.py to work again in standalone mode Internal libtiff: resync with upstream CMake: enhance build_hints.rst and various fixes/enhancements cmake: remove tabulations in .cmake files CMake: declare option GDAL_USE_PUBLICDECOMPWT CMake: add support for external LERC library (fixes #5088) cmake: FindSPATIALITE: fix for Conda windows cmake: FindMySQL: fix for Conda windows CI: enable Spatialite on Conda CMake windows builds msys2 mingw64 build: fix warning about BLOB being defined in a system header msys2 mingw64 build: fix warning about type mismatch in CPLDebug() and unused argument msys2 mingw64 build: use ll formatter instead of I64 msys2 mingw64 build: fix weird warning about string overflow msys2 mingw64 build: avoid -Wnull-dereference warnings msys2 mingw64 build: avoid warning about invalid reinterpret_cast<> msys2 mingw64 build: avoid warnings in cpl_vsil_crypt.cpp and cpl_sha256.cpp msys2 mingw64 build: attempt fixing warning about CPL_FRMT_GIB in ogrosmlayer.cpp msys2 mingw64 build: fix warning about CPL_FRMT_GIB in NAS driver CI build-windows-msys2-mingw: turn warnings as errors AAIGRID: add support for 'null' as can be generated by D12 software (fixes #5095) GTiff: fix DISCARD_LSB with nodata value (fixes #5097) Internal libtiff: resync with upstream Internal libjson: update to 0.15.0 Internal libjson: GDAL specific changes Internal libjson: compiler warning fixes Add .zenodo.json file (#5101) Add FUNDING.yml .github/FUNDING.yml: add link to numfocus donate to GDAL [ci skip] OGRSpatialReference::importFromEPSG(): document OSR_USE_NON_DEPRECATED=NO configuration option (refs #5106) gdalsrsinfo: emit message when replacement of deprecated CRS occurs, with hint to set OSR_USE_NON_DEPRECATED=NO configuration option to avoid that (fixes #5106) cmake: check linux/fs.h presence CMake: fix MACOSX_FRAMEWORK setting CMake: remove unused options, mark some as advanced CMake: enable -ftrapv for debug builds CMake: detect __builtin_bswap32 / __builtin_bswap64 CMake: add build support for dummy NULL driver CMake: add support for libhdfs (for /vsihdfs) CMake: add missing detection of a few functions fuzzers/build.sh: build Poppler with sanitize flags to avoid https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43668 [ci skip] CMake: add GDAL_AUTOLOAD_PLUGINS option CMake: enable -DDEBUG for debug builds Add a prefix/lib/gdalplugins/drivers.ini file to control registration order GTiff: RoundValueDiscardLsbUnsigned(): fix potential unsigned overflow (CID 1469012) Internal libjson: add assertion to avoid CID 1469011 Internal libjson: fix printf formatter (CID 1448587) RFC 84: modify planned schedule according to proposal of https://lists.osgeo.org/pipermail/gdal-dev/2022-January/055253.html CI ASAN: port to Ubuntu 20.04 (fixes #5124) OGREnvelope: add == and != comparison operators WMTS: disable clipping with TileMatrixSetLimits by default when using layer extent and add open options to control that behavior (fixes #4461) WMTS: document open options (fixes #4471) VSISync() onto /vsis3/: allow x-amz- headers to be specified for object creation /vsis3/: ignore object class DEEP_ARCHIVE in addition to GLACIER, and add CPL_VSIL_CURL_IGNORE_STORAGE_CLASSES to configure which object classes should be ignored netCDF: lint: better variable name gdalbuildvrt: change logic to check homogenous number of bands (refs #5136) Python bindings and utilities-as-lib: propagate warnings to custom error handler when UseExceptions is on (fixes #5136) gdalbuildvrt: add a -strict/-non_strict flag, and in strict mode consider non-existing datasets as a failure (fixes #4755) Add DOI to README.md and CITATION [ci skip] CITATION: add doi to BibTex entry [ci skip] Doc: add DOI to about page byndataset.cpp: whitespace fixing [ci skip] BYN: remove validation of nTideSys and nPtType fields netCDF: add WRITE_GDAL_VERSION and WRITE_GDAL_HISTORY creation option Updates for IJG libjpeg-9e /vsigs/, /vsiaz/, /vsiadls/: allow x-goog- / x-ms-/ headers to be specified for object creation Replace reference to MIT/X or X/MIT to MIT (fixes #5154) CMake: use Poppler_ for variable names Update swig/csharp/dll_template.csproj CMake CSharp: fix dependency rules so that a second 'make' without any change doesn't result in .dll and sample files to be regenerated Add RFC 85: Policy regarding substantial code additions (#5128) CI: fix crash on Windows Conda CMake builds /vsis3/: make sure file properties of /vsis3_streaming/foo are invalidated when /vsis3/foo ones are CI: disable build-windows-conda-csharp-only job OGR MEM driver: set GDAL_DCAP_FIELD_DOMAINS and GDAL_DMD_CREATION_FIELD_DOMAIN_TYPES metadata items (instead in the raster MEM driver) [master only] TGA: fix reading images with runs crossing scanlines (fixes #5168) CMake: MBTiles: fix for external Zlib Doc: add L3Harris/Harris Geospatial Solutions as bronze sponsor Add forgotten file CMake: fix build with libxml2 that may require Iconv CMake FindHDF4: fix how we look for ws2_32 nitf.py: workaround issue with Conda Windows and libjpeg-9e gdal.i: remove duplicated GDAL_RMS constant netCDF: add a VARIABLES_AS_BANDS=YES/NO open option netCDF: allow update mode of raster datasets netCDF: implement SetMetadataItem()/SetMetadata() netCDF: avoid warnings when CreateCopy() a non-georeferenced dataset, and opening a 1x1 non-georeferenced dataset Update version numbers to 3.5.0/3.5.0dev HOWTO-RELEASE: add a step to bump version numbers for the -dev version [ci skip] Driver manager: make sure oMapNameToDrivers.size() == nDrivers when querying a non existing driver GDALDriverManager::ReorderDrivers(): add assertion ogr_gpkg.py: remove temporary file CMake: fix minimum build with internal zlib, libjpeg and libtiff CMake: swig/python/setup.py.in: use GDAL_LIB_OUTPUT_NAME variable CI CMake: add a minimum build configuration GPKG: when adding a 'epoch' column to gpkg_spatial_ref_sys, one must use the 'gpkg_crs_wkt_1_1' extension instead of 'gpkg_crs_wkt' Travis CI: fix getting get-pip.py for Python 3.6 GPKG: DeleteField(): use ALTER TABLE ... DROP COLUMN if sqlite >= 3.35.5, and run foreign_key_check only if foreign_keys is ON (fixes qgis/qgis#47012) Move GDALGeoPackageDataset::PragmaCheck() to OGRSQLiteBaseDataSource SQLite: DeleteField(): use ALTER TABLE ... DROP COLUMN if sqlite >= 3.35.5, and run foreign_key_check only if foreign_keys is ON (refs qgis/qgis#47012) GPKG: AlterFieldDefn(): use ALTER TABLE ... RENAME COLUMN when only renaming is to be done and sqlite >= 3.26.0, and do not run integrity_check in that situation (fixes qgis/qgis#47012) SQLite: AlterFieldDefn(): use ALTER TABLE ... RENAME COLUMN when only renaming is to be done and sqlite >= 3.26.0 SQLite/GPKG: avoid CPLDebug() error message when executing ExecuteSQL("PRAGMA table_info(..)") GPKG: intercept 'ALTER TABLE table RENAME COLUMN src_name TO dst_name' and 'ALTER TABLE table DROP COLUMN col_name' CMake: implement documented GDAL_USE_TIFF switch, and avoid copy&paste patterns CheckDependentLibraries.cmake: remove unsued HAVE_XMLPARSER variable CheckDependentLibraries.cmake: use gdal_check_package() for OpenSSL CMake: make all dependencies found disableable CMake: add a GDAL_USE_EXTERNAL_LIBS=OFF option to disable all external libs build_hints.rst: document missing GDAL_USE_ variables CMake: rename GDAL_ENABLE_FRMT_ variables to GDAL_ENABLE_DRIVER_ and OGR_ENABLE_ to OGR_ENABLE_DRIVER_ CMake: fix use of undefined variables CMake: rename GDAL_JPEG12_SUPPORTED to GDAL_USE_LIBJPEG12_INTERNAL CMake: fixes for cmake/template/gdal_def.h.in CMake: warn when turbojpeg is used frmts/netcdf/CMakeLists.txt: cleanup CMake: allow to build HTTP driver as a plugin CMake: rename GDAL_USE_LIB{foo}_INTERNAL to GDAL_USE_{foo}_INTERNAL CMake: fix 'export ordinal too large' error with mingw and default build type netcdfdataset.cpp: use std::tie() GTIFPrintDefn(): avoid potential nullptr deref. CID 1469240 plmosaicdataset.cpp: explicitly ignore ret value of VSIMkdir() (CID 1469242) Internal libjson: complement to fix CID 1448587 ILI IMDReader: fix various potential crashes on invalid output (CID 1469239, 1469241, 1469243, 1469244, 1469245) CMake: fix build of DDS driver as a plugin frmts/drivers.ini: fix registration order of GNM drivers CMake: fix installing drivers.ini in gdalplugins directory of build tree Delete .DS_Store [ci skip] cmake_builds.yml: disable ctest on build-windows-conda job Revert "cmake_builds.yml: disable ctest on build-windows-conda job" cmake_builds.yml: build-windows-conda: disable HTTP driver (as before 6b05d80f01c86880e5729fb59243dba8a1f48e49) GPKG: add a NOLOCK=YES option to open a file without any lock (for read-only access) (helps fixing qgis/qgis#23991, but requires QGIS changes as well) GTiff: fix exposing WEBP_LOSSLESS option CMake: GTiff: fix exposing WEBP_LOSSLESS option autotest/pymod/webserver.py: fix to not start a server on a port already occupied on Windows gdalhttp.py: skip test_http_ssl_verifystatus() on build-windows-conda Revert "cmake_builds.yml: build-windows-conda: disable HTTP driver (as before 6b05d80f01c86880e5729fb59243dba8a1f48e49)" CMake: make it use CMake config files in priority to find PROJ CMake: use gdal_target_link_libraries() to link PROJ CMake: align signature of gdal_target_link_libraries() on standard target_link_libraries() CI: ogr_gmlas.py: skip HTTP related tests on build-windows-conda Remove doc/.azure-pipelines.yml given that we now have a github action for that [ci skip] CMake: generalize use of gdal_target_link_libraries() to non-driver object libraries doc_build.yml: fix MVT writing: using MakeValid() when possible on polygon output... Add missing const references in argument passing netcdfdataset.cpp: remove always true conditions MapInfo .tab writing: correctly detect datum when creating a layer from a WKT2 CRS string (fixes #5217) FileGDB: workaround a crash involving binary field and CDF datasets CMake: fix minimal build without JPEG driver CI: adapt for windows-latest environment which is now windows-2022, and explictly use that version autotest/cpp: relicence LGPL licensed files to MIT except one exception CMake: fix 'Invalid escape sequence' warning on Python bindings install on Windows CI CMake: turn on -Werror=dev SWIG: add gdal.DataTypeUnion() README.md: update badge for docs [ci skip] netCDF: add a IGNORE_XY_AXIS_NAME_CHECKS=YES open option (refs qgis/QGIS#47158) gdal_calc.py: support wildcard for filenames, and use 3D arrays instead of list for multiple filenames per alpha Warped VRT opening: do not open the source dataset with GDAL_OF_SHARED GDALAutoCreateWarpedVRT(): clarify source dataset ownership with example bash-completion: fixes and update gdal-bash-completion.sh Rename cmake/helpers/GdalSetTestEnv.cmake to GdalSetRuntimeEnv.cmake and use it in autotest/CMakeLists.txt scripts/setdevenv.sh: add /home/even/gdal/gdal/swig/python/gdal-utils to PYTHONPATH for autoconf builds CMake: add a 'completion' target and install bash completion scripts (fixes #5232) CMake: install python: fix DESTDIR support CMake: make URL of PublicDecompWT SDK configurable and use my github clone for CI since eumetsat gitlab is down /vsitar/: fix reading .tar.gz files when the size of the uncompressed .tar file is a multiple of 65536 bytes (fixes #5225) GDALAutoCreateWarpedVRT doc: better code example formatting VRTWarp: fix serialization of relativeToVRT=1 when mix of relative and absolute paths for source and VRT (refs #4983) VRT (non warped): allow setting VRT description after CreateCopy('', src_ds) VRT (non warped): fix serialization of relativeToVRT=1 when mix of relative and absolute paths for source and VRT CPLGetCurrentDir(): use _wgetcwd() on Windows to get a UTF-8 filename CMake: remove getcwd related settings in MSVC case configure.cmake: cleanups in MSVC case Rasterization of polygons: avoid underflow/overflow of output data type Rasterize: use rounding to integers of floating-point value, and handle +/- inf for floating point rasters Delete .github/LICENSE.TXT [ci skip] CMake: internal libtiff: fix build with external libjpeg and internal libjpeg12 Python bindings: commit regenerated files [ci skip] tif_jxl: bring back support for v0.6.1 since there isn't yet any libjxl release with JxlEncoderFrameSettingsSetOption() JP2KAK: fix build on Windows (fix ed327928c89841a81023ad67165f6c428ead7d50) [ci skip] frmts/gtiff/libtiff/CMakeLists.txt: fix build without JPEG GeoRaster driver: fix build without JPEG CMake: GeoRaster: fix build without JPEG CMake: tidy linking of Windows system libraries ogrlineref.cpp: fix cppcheck warning CMake: apps/CMakeLists.txt: avoid useless liking of external libraries in binaries Internal libpng: fix memleak on corrupted file. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44486 GRIB: avoid harmless unsigned integer overflow. master only. fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43652 Build remove message about experimental status of CMake system, and add messages about deprecation of autoconf&nmake (fixes #5171) Remove deprecated testepsg utility (fixes #3993) /vsigs/: fix upload of files > 4 MB in HTTP 1.1 (fixes #5267) GTiff: SRS reader: avoid floating-point division by zero on invalid file. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44502 CMake: fix build with BUILD_APPS=OFF (fixes #5275) CMake: make get_dep_libs() work with several library names passed to a _LIBRARY variable (fixes #5272), and also fix variable inversion for INTERFACE_LIBRARY drivers.ini: add NUMPY driver [ci skip] gdal_calc.py: raise error when overwriting behavior is apparently wished but not specified (fixes #5270) CMake: fix building without GNM (fixes #815) CheckDependentLibraries.cmake: fix CMAKE_MSVC_RUNTIME_LIBRARY value for Armadillo detection FindHDF4.cmake: fix version fetching, which caused a weird 'CMake Error: -D must be followed with VAR=VALUE' later in FindNetCDF... FindNetCDF.cmake: add needed inclusion CMake: use the policy_max option of cmake_minimum_required() to avoid turning on manually policies JP2OpenJPEG: add a STRICT=YES/NO open option to allow decoding some broken files (required OpenJPEG 2.5) OGRSpatialReference::SetFromUserInput(): make it work with 'IAU:XXXX' and 'IAU:2015:XXXX' (when using PROJ >= 8.2) autotest/gcore/vsis3.py: remove temporary file [ci skip] ECW: fix non-nearest upsampling on multi-band datasets (fixes #5288) CMake: do not use PATH when finding a package config file GRIB: degrib: fix use of unitialized memory on some datasets (refs #5290) GRIB: fix write heap-buffer-overflow when using 'split and swap column' mode when the split column is not equal to width / 2 (fixes #5290) netCDF: recognize x/y axis from GMT generated files as geospatial axis (fixes #5291, fixes qgis/QGIS#47158, fixes qgis/QGIS#45704) Revert "CMake: do not use PATH when finding a package config file" Internal libtiff: resync with upstream GTiff: early checks for PREDICTOR settings, and update internal libtiff to support PREDICTOR=2 for 64-bit samples (fixes rasterio/rasterio#2384) Python bindings: remove all traces of distutils (refs https://github.com/OSGeo/gdal/pull/5296#issuecomment-1037460842) gdal.cmake: remove 'set(CMAKE_MACOSX_RPATH ON)' as this is the default behavior with CMake 3.0 VRT: Log10PixelFuncHelper(): micro optimization in complex case Remove outdated TODO comment [ci skip] Fix doc building with breathe 4.33.0 Unix/CMake build: prefix internal libpng symbols with gdal_ (fixes #5303) CMake: add gdal_add_vendored_lib() to ease using internal libs, with potential renames, and use it CMake: rename by default internal zlib symbols CMake: mark RENAME_ options as advanced CMake: rename by default internal giflib symbols OGRSpatialReference::SetFromUserInput(): do not emit error message on unrecognized string when ALLOW_FILE_ACCESS=NO (fixes Toblerity/Fiona#1063) gdalwarp: avoid useless use of CHECK_WITH_INVERT_PROJ when computed bounds are at the 'edges' of the geographic domain warper: better guess output bounds when warping from a rotated pole projection that include poles Doc: fix python_samples.rst (fixes #5205) Doc: fix previous commit Overview generation: fix Cubic resampling on boundaries between valid and transparent areas (fixes #5016) CI: disable build-windows-msys2-mingw (refs #5317) OGR SQLite dialect: avoid being confused by an attribute and geometry field of same name CSV layer: when the .csvt indicates WKT and in default KEEP_GEOM_COLUMNS=YES mode, prefix the geometry field name with 'geom_' GeoPackage: add ST_MakeValid() SQL function if not linking against Spatialite or if Spatialite lacks ST_MakeValid() Shapefile writer: avoid considering rings slightly overlapping as inner-outer rings of others (refs #5315) CODE_OF_CONDUCT.md: point to website code of conduct [ci skip] Shapefile writer: fix perf issue when writing multilinestring with lots of parts (fixes #5321) GTiff: remove limitation to 32,000 bytes when writing the GDAL metadata tag (fixes #4116) Shapefile writer: consider rings at non-constant Z as outer rings (fixes #5315) Disable cmake-format CMake: avoid error when importing broken expat::expat target on Ubuntu 21.10 (fixes #5324) COG: clarify that ALIGNED_LEVELS effect starts at 2 or more (fixes #5336) CEOS2 driver: avoid false positive from msys mingw64 about 'array subscript 'double[0]' is partly outside array bounds of 'int[1]' [-Werror=array-bounds]' MSSQLSpatial: fix warnings and build errors on msys mingw64 with msodbcsql.h CI: fix build-windows-msys2-mingw builds Add a OSR_DEFAULT_AXIS_MAPPING_STRATEGY configuration option autotest: updates to enable it to pass with OSR_DEFAULT_AXIS_MAPPING_STRATEGY=TRADITIONAL_GIS_ORDER CMake: fix handling of geotiff-config.cmake COG driver: output exactly square pixels when using -co TILING_SCHEME (fixes #5343) warp: ComputeSourceWindow(): round source coordinates when very close (refs #5343) PDF: fix build against Poppler > 22.2 fuzzers/build.sh: build against Poppler > 22.2 warper: avoid using any resampling kernel when doing just a subsetting of the source image (alignement on pixel boundaries) (refs #5345) libtiff: resync with upstream CSLTokenizeString2(): make it work with strings > 2 GB GTiff: code linting regarding compression numeric code <--> string conversions CMake: add a warning message if a gcc/clang single config build results in a non-optimized build Docker README.md: update versions [ci skip] CI: re-enable libxml2 for build-windows-msys2-mingw gdal_rpc.cpp: fix warning about untrusted allocation size (CID 1475177) internal libqhull: initialize a variable (CID 1475189) gdalmdiminfo_lib / gdalmdimtranslate_lib: try to avoid false positive warning about nullptr deref (CID 1475122, 1475115, 1475134) gdalmdimtranslate_lib: avoid potential nullptr deref (CID 1475176) GRIB: avoid false positive warning about nullptr deref (CID 1475140) GRIB: avoid false positive warning about poDS being tainted... (CID 1475169) geotiff.cpp: avoid warnings about overflow_before_widen (CID 1475105, 1475116, 1475204) BAG: avoid warnings about overflow_before_widen (CID 1475131, 1475142) netcdfdataset.cpp: avoid false positive warning about nullptr deref (CID 1475193) netcdfsgwriterutil.cpp: silence false positive warning about null dereferenced (CID 1475171) PCIDSK SDK: silence false positive warning about string non null terminated (CID 1475184) PCIDSK SDK: avoid false positive warning about resource leak (CID 1475126) PCIDSK SDK: silence false positive warning about string non null terminated (CID 1475183) PCIDSK SDK: avoid false positive warning about resource leak (CID 1475145) PCIDSK SDK: avoid false positive warning about uninitialize members (CID 1475173) PCIDSK SDK: avoid false positive warning about uninitialize members (CID 1475190) PDF: avoid using a moved object (CID 1475123) PDF: avoid false positive warning about double free (CID 1475205) gdalrasterband.cpp: avoid false positive warning about overflow_before_widen (CID 1475108, 1475110, 1475125, 1475132, 1475191) overview.cpp: avoid false positive warning about overflow_before_widen (CID 1475127, 1475168, 1475202) GDALRasterBand::IRasterIO(): avoid potential integer overflow (CID 1475186) RawRasterBand::IRasterIO(): avoid potential integer overflows (CID 1475151, 1475198) GNMRule: avoid uninitialized members in constructors (CID 1475155, 1475182) libopencad: avoid false positive warning about uninitialized mat variable (CID 1475187) CSV: avoid coverity scan warning about untrusted value (CID 1475200) DXF: avoid potentially uninitialized members in DXFMLEADERLeader (CID 1475164) FileGDB: avoid warning about unchecked return value (CID 1475162) GPKG: avoid potential nullptr deref (CID 1475107, 1475124) MITAB: fix memleak in error code path (CID 1475129) MVT: avoid false positive warning about nullptr deref (CID 1475137, 1475148) OpenFileGDB: avoid false positive warning about nullptr deref (CID 1475133, 1475196) SQLite: avoid potential nullptr deref (CID 1475158, 1475112) cpl_time.cpp: avoid overflow_before_widen warning (CID 1475180) Avoid warnings about curl_easy_setopt() return value not being checked LercLib: fix warning about comparison to uninitialized m_nCols member in copy constructor (CID 1475121) SQLite: fix crash when doing select load_extension('libgdal.so') from a statically linked sqlite3 console application CMake: fix error in gdal_generate_config() when CMAKE_C_IMPLICIT_LINK_LIBRARIES is empty CI: make ctest actually do something with cmake 3.10.2 on Ubuntu configuration SQLite: add a OGR_SQLITE_LOAD_EXTENSIONS configuration option gdal2tiles: detect write error when creating tiles, and allow outputing to /vsi filesystems (fixes #3382, fixes #5370) fuzzers/build.sh: fix ossfuzz build [ci skip] Python bindings: make sure GetUseExceptions() doesn't clear the error state (fixes #5374) gdal2tiles: display overview progress bar only if there are overviews (master only) swig/python/setup.py.in: fix for standalone builds CI: test gdal-utils package (fixes #5344) SQLite: fix VirtualShape support with spatialite 5.0.1 or older and sqlite 3.38.0 (unbreaks CI CMake MacOS target) SQLite: amend code comment [ci skip] CI: enable ccache on other CMake configs CMake: fix dependencies of pytest_runner gdal.cmake: emit warning for Ninja MSVC builds without explicit CMAKE_BUILD_TYPE [ci skip] CI build-windows-conda: add blosc dependency [ci skip] Internal libtiff: resync for OSGeo/libgeotiff#66 fix (which doesn't seem to affect GDAL) CI: fix and enable msys2 tests CI build-windows-msys2-mingw: use default msys2 shell for running ctest Typo fixes GeoJSON: report OFSTJSON subfield type for properties we can't map to a native OGR type, and better handli…
What is the bug or the crash?
Renaming a column's name of a large gpkg file (~18 GB) using the DB Manager takes about 30 minutes.
Example picture with a different file:
![image](https://user-images.githubusercontent.com/67278094/151136316-51bea01b-1a09-4284-ac8e-e96610aa5963.png)
Having a quick look at gpkg spec at https://www.geopackage.org/spec/ makes seem gpkg to be hierarchically structured such that I assume not every column name of each value needs to be rewritten individually which would explain such long processing times. If a gpkg is well structured I wonder about such long processing times. Or is the reason that the file is instantly rewritten entirely at the moment of the edit?
Doing the same rename in a PostgreSQL Database takes about 2 seconds.
Steps to reproduce the issue
Load gpkg file from https://drive.google.com/file/d/1rEBI600wM54Ztm9CbK4-7i6YUKQq1uNH/view?usp=sharing into the DB manager and change one column's name using the "edit column" button.
Example picture with a different file:
![image](https://user-images.githubusercontent.com/67278094/151137284-0991ffff-84fa-4d44-89b4-9138e85d557b.png)
After hitting the OK button the DB Manager and QGIS are stalled for half an hour
Versions
QGIS version
3.22.3-Białowieża
QGIS code revision
1628765
Qt version
5.15.2
Python version
3.9.5
GDAL/OGR version
3.4.1
PROJ version
8.2.1
EPSG Registry database version
v10.041 (2021-12-03)
Compiled against GEOS
3.10.0-CAPI-1.16.0
Running against GEOS
3.10.2-CAPI-1.16.0
SQLite version
3.35.2
PDAL version
2.3.0
PostgreSQL client version
13.0
SpatiaLite version
5.0.1
QWT version
6.1.3
QScintilla2 version
2.11.5
OS version
Windows 10 Version 2009
Active Python plugins
GeoCoding
2.18
GroupStats
2.2.5
HCMGIS
21.8.28
mmqgis
2021.9.10
Qgis2threejs
2.6
QuickOSM
2.0.0
quick_map_services
0.19.27
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.5
processing
2.12.99
sagaprovider
2.12.99
Supported QGIS version
New profile
Additional context
No response
The text was updated successfully, but these errors were encountered: