renaming/deleting columns takes a lot of time on large vectors #47012
Assignees
Labels
Bug
Either a bug report, or a bug fix. Let's hope for the latter!
Data Provider
Related to specific vector, raster or mesh data providers
DB Manager
Relating to the DB Manager core plugin
Upstream
Needs changes in an upstream library (like Qt, Proj, GDAL, ...)
Comments
tschmetzer
added
the
Bug
|
I'm not totally sure that this is only a DB Manager issue. For example renaming/deleting a column with OGR from a such large vector it takes a fraction of a second.
while is painfully slow in QGIS (for example deleting the same column from the layer properties > fields). There is large margin for improvement here, and as far as I remember this is something that affects QGIS since... forever. |
gioman
changed the title
gioman
added
the
Data Provider
gioman
changed the title
20 minutes and counting after giving the command to save changes after having deleted a column (within the "fields" tab in layer properties) in the provided sample vector. Ogrinfo > almost instantaneous. |
|
Wait, to add insult to injury: |
rouault
added
the
Upstream
|
OSGeo/gdal#5193 will improve thins in particular for renaming that becomes instantenous. |
|
@rouault thanks! this is on my 10 years old laptop and that 18GB GPKG. Dropping a column is pretty fast! (sorry for the watermark in the cast, but in this period I must use a Windows machine) Recording.4.mp4 |
that's totally unexpected. The whole table should be mostly rewritten (from https://www.sqlite.org/lang_altertable.html "The DROP COLUMN command removes the named column from the table, and rewrites its content to purge the data associated with that column"). Which SQLite version is this: try |
|
|
@gioman and is the content of the records of the table after column deletion correct ? The PO column should have values at 3333.3321 |
@rouault no! you are right, the PO column got the values of the deleted column... |
ok, that is probably related to what I mentioned in https://trac.osgeo.org/osgeo4w/ticket/726 : "It was available in 3.35.0 but according to https://www.sqlite.org/changes.html corruption could occur until 3.35.5 " |
rouault
added a commit
to rouault/gdal
that referenced
this issue
Jan 30, 2022
…5.5, and run foreign_key_check only if foreign_keys is ON (fixes qgis/QGIS#47012)
rouault
added a commit
to rouault/gdal
that referenced
this issue
Jan 30, 2022
….35.5, and run foreign_key_check only if foreign_keys is ON (refs qgis/QGIS#47012)
rouault
added a commit
to rouault/gdal
that referenced
this issue
Jan 30, 2022
…enaming is to be done and sqlite >= 3.26.0, and do not run integrity_check in that situation (fixes qgis/QGIS#47012)
clrpackages
pushed a commit
to clearlinux-pkgs/gdal
that referenced
this issue
Jun 11, 2022
Alan D. Snow (1):
netCDF: always use WKT when found, without comparing with CF params (#4734) (fixes #4725)
Alan Snow (2):
REF: fix variable name dfMultFactorVerticalShift
DOC: Added APPLY_VERTICAL_SHIFT & APPLY_VERTICAL_SHIFT to GDALWarpOptions::papszWarpOptions docstring
Alan Thomas (1):
DXF: Fix long line handling in edge case
Alexander Nehrbass (1):
XYZ-driver: Fix incorrect failure to open ASCII-file due to floating point comparison
Andrea Giardini (1):
Google cloud - Support type=user JSON file for authentication
Andrea Giudiceandrea (13):
GPKG: fix GDAL_DMD_HELPTOPIC
GPKG: add redirection to correct doc page
GPKG: fix doc redirection
Docs: gdal_rasterize: fix single/double quotes for '-where' parameter (#5241)
Add decl_configoption for GDAL_NUM_THREADS in gdal_grid docs
Add decl_configoption in jp2openjpeg docs
Add decl_configoption in sentinel2 docs
Add decl_configoption in XLSX driver docs
Add decl_configoption in XLS driver docs
Docs: fix a typo in gdal_calc
Docs and help strings: add missing -pct option description in pct2rgb and fix some typos in pct2rgb and rgb2pct (#5447)
Fix gdal_grid link in XYZ driver docs
Fix gdal_edit link in ECW driver docs
Andreas Degwerth (Airbus) (1):
NITF: RPF.toc Skip plausibility check for Overviews and Legends; disable some checks for polar zones (#5654)
Andrew C Aitchison (2):
CMake: make OGDI driver build on Ubuntu 21.10 (#4905) (fixes #4817)
OGREnvelope: ignore fp warnings for operator== like IsInit()
Antonio Valentino (1):
VRT: add div, polar, exp pixel functions. improvements in sum, mul, inv (#5308)
Bas Couwenberg (1):
Update BAG 2.0 Extract URLs
Björn Harrtell (1):
FlatGeobuf: If CRS WKT detected to be non UTF-8, force it to ASCII (#5062)
Brendan Jurd (1):
PG: skip all leading whitespace in SQL statements. (#4787)
C41eb (1):
Python bindings: Change GetFieldAsBinary to use VSIMalloc. (#4774)
Chiara Marmo (1):
Migrate doc building to github action workflows (#5182)
Chris Barker (1):
Updated install instructions
Chris Cantey (1):
Update python.rst (#5658)
Cédric Traizet (1):
Fix broken link for the news page
Daniel Baston (1):
gdalwarp: Test and document mode behavior in case of ties
Dominik Michael Rauh (1):
port/cpl_recode_iconv.cpp: fix invalid cast error with uclibc (#5684)
Dominik Nussbaumer (1):
rdb: enable building rdb driver via CMake (#4816)
Egor Pugin (1):
Initialize jxl structures with zeros. Fixes VS17.0 build error. Closes #4842.
Even Rouault (1202):
.gitignore: remove CMakeLists.txt and .cmake files
autotest/CMakeLists.txt: fix for cmake < 3.14
charls: use 'CharLS::charls' case
test_cpl.cpp: avoid redefinition of GDAL_COMPILATION
CMake: fix CONFIG_DATA to not include GDAL version
port/CMakeLists.txt: restrict exported headers to those of port/GNUmakefile
CMake: install headers in prefix/include, for compat with current practice
CMakeLists.txt: remove explicit cmake_policy() that are enforced by the 3.10 min requirement, and add comments for remaining ones
CMake: make us resilient to absence of autotest directory, as found on tarballs
CMake: detect issue with conflicting cpl_config.h
CMake: only set POSITION_INDEPENDENT_CODE ON for BUILD_SHARED_LIBS=ON
frmts/CMakeLists.txt: make more formats optional
CMake: change cpl to an object library, to help with static builds
CMake: make bindings build optional
update_copyright.py: move under scripts/
port/CMakeLists.txt: simplify unity build setting, and better linking properties for cpl library
test_gdal.cpp: do not crash if PCIDSK driver not available
mrf.py: fix when driver not available
cmake/template/pytest.ini.in: add gdrivers/data to -norecursedirs
frmts/CMakeLists.txt: JPEG is required for TIFF driver
cmake: make gdal_dependent_format/ogr_dependent_driver default to OFF if GDAL_BUILD_OPTIONAL_DRIVERS/OGR_BUILD_OPTIONAL_DRIVERS is OFF
CMake: make a few more formats optional
gcore/CMakeLists.txt: fix OGR_ENABLE_SQLITE=OFF builds
CMake: fix MRF build
CMake: remove possibility to build against external PCIDSK SDK, as it is not commonly found
.cmake-format.py: import from https://github.com/miurahr/cmake4gdal
CMake: apply cmake-format -i
CI: add .github/workflows/cmake-lint.yml (adapted from https://github.com/miurahr/cmake4gdal)
alg/: does not depend on SFCGAL
CMake: add explicit linking to gmpxx for SFCGAL 1.3.7
CMake: fix paths to now obsolete gdal/ subdirectory
CMake: fix gdal-config --libs generation for the 'dl' library added through
CMake: move inclusion of libcsf subdirectory to frmts/pcraster/CMakeLists.txt
gdal.cmake: tune how we link gdal to GDAL_LINK_LIBRARY
CMake: only apply -msse/-mavx to gdalgridsse.cpp/gdalgridavx.cpp
CMake: fix internal libjpeg build
CMake: fix CAD building on Windows
CMake: default to GDAL_HIDE_INTERNAL_SYMBOLS=ON
CMake: remove -DBIGTIFF_SUPPORT
CMake: fix URL to libtiff project
CMake: simplify a bit ZSTD TIFF conf
CMake: autotest: fix copying of autotest content on Windows
CMake: fixes related to internal libtiff/libgeotiff renaming (enable it by default)
Remove obsolete tests and uses of dbmalloc.h
Remove STDC_HEADERS setting / checks
CMake: improve logic to generate gdal_version.h, and remove the sha1sum from the GDAL_VERSION
CMake: add trick w.r.t gdal_version.h to minimize the amount of rebuild
CMake: fix build issue with ninja and Python bindings
CMake: remove dbmalloc / stdc_headers related stuff with MSVC
CMake: remove unused TARGET_LINK_LIB
CMake: fix linking to internal libjson in ogr component
CMake: turn all STATIC libs to OBJECT libs, and add their link libraries through a global property
cmake: fix gdal-config CONFIG_DEP_LIBS value for -lm
CMake: fix FindPoppler.cmake for Windows / conda
CMake: fix include order for GTiff driver
CMake: make alg depend on generate_gdal_version_h
CMake: generate_gdal_version_h.cmake: avoid error when git not found
.github/workflows/cmake_builds.yml: disable all jobs but build-linux-ubuntu-focal
CMake: fix build of GPKG driver
CMake: fix build of autotest/cpp on mingw64
.github/workflows/cmake_builds.yml: enable build-windows-conda
CMake: autotest/cpp: do not modify PATH as it doesn't work with MSVC
CMake: only run test-copy-words on build-windows-conda config
cmake: FindNetCDF: fix retrieval of NETCDF_HAS_xxxx features
CMake: rik: fix inclusion of Zlib
CMake: internal libjson: set _XOPEN_SOURCE to 700
CMake: fix shapelib build regarding SAOffset
CMake: geoconcept: add MSVC warning suppression
CMake: internal libtiff: tune include order
CMake: avoid incompatible configuration regarding internal libtiff/libgeotiff
CMake: emit warning about experimental status
CMake: add extra warnings from configure/nmake
CMake: add -Werror to build-linux-ubuntu-focal job
CMake: fix NETCDF_HAS_NC4 detection on Windows
CMake: fix use of wrong variable in GdalVersion.cmake
CMake: add missing dependencies to generate_gdal_version_h
CMake: replace CMAKE_[SOURCE/BINARY]_DIR witch PROJECT_[SOURCE/BINARY]_DIR to be able to add_subdirectory(gdal)
CMake: test building as a add_subdirectory()
CMake: fix LIBKML finding
CMake: disable libkml on build-windows-conda
CMake: add JPEG12 support
CMake: add gdal_create and gdal_viewshed
cmake/modules/DefineFindPackage2.cmake: fix typo
cmake/modules/DefineFindPackage2.cmake: other typo fixes
CMake: remove BPG detection
CMake: CheckDependentLibraries.cmake: typo fix
CMake: slightly clearer logic regarding rasterlite2
CMake: avoid rebuild of tif_jpeg* files
Add scripts/reformat.sh: to reformat CMakeLists.txt files for now
CMake: remove trace of BPG driver
CMake: fix HINSTS typo, and also fix Rasterlite2 package config variable names
CMake: rename ENABLE_LTO as ENABLE_IPO
CMake: idrisi: remove unappropriate plugin test
CMake: remove USE_CPL option
CMake: make it possible to disable a lot of found dependencies
CMake: Doc: initiate a documentation page (WIP !)
CMake: revise how builtin vs plugin drivers are declared
CMake: fixes regarding plugin build
CMake: fix minimal build
CMake: enable MITAB driver
CMake: adjust how to specify a driver name for option names when it is not the same as the subdirectory
CMake: fix JPEG2000/Jasper driver
CMake: GTiff: fix minimal build
CMake: pytest.ini.in: add deprecation driver suppressions
CMake: make OpenCL opt-out, and disable it by default
CMake: add libdeflate support
CMake: add LZ4 support
CMake: FindLZ4.cmake: although look for liblz4 (for Windows Conda)
CMake: FindLibLZMA: remove unneeded extra function checks (for Windows Conda)
CMake: gtiff/mrf: modify to have internal library includes first
CMake: add blosc support
CMake: fix getting LibLZMA,INTERFACE_INCLUDE_DIRECTORIES
CMake: add TileDB support
CMake: FindTileDB.cmake: fix warning with cmake >= 3.19
CMake: FindTileDB.cmake: remove useless variable
CMake: improve doc
CMake: doc: use 'compilers' package
CMake: use internal shapelib by default
CMake: remove detection of SDE, since the drivers have already been removed
CMake: make CharLS package disableable
CMake: make CFITSIO disableable
CMake: make Armadillo disableable
CMake: alg/CMakeLists.txt: remove useless target_compile_definitions
CMake: changes to have the same set of headers files installed as for autoconf builds
CMake: reorder inclusions so that internal libraries are included first
CMake: cleanup and fix setting INST_DATA define to find resource files
CMake: fix issues spotted by cmake --warn-uninitialized
CMake: fix Windows build
CMake: remove FindTileDB.cmake that is not needed
CMake: define CPL_DISABLE_DLL for static MSVC builds
CMake: remove unexisting cmake 3.9 directory
CMake: revise version numbers in libgdal.so* files to match libtool convention
CMake: autotest/CMakeLists.txt: correctly use CMAKE_HOST_SYSTEM_NAME
CMake: suppress warnings for Windows builds of internal libjson
CMake: add warnings suppression for MSVC on internal libz and libtiff
CMake: add warnings suppression for MSVC on internal libjpeg
CMake: make MSVC debug postfix customizable
CMake: number of fixes and improvements for Python bindings
CMake: PCIDSK: enable JPEG support and fix build as plugin
CMake: PCRaster: enable build as plugin
CMake: fix -fvisibility=hidden
CMake: CAD: fix build as plugin
CMake: make PNG driver PLUGIN_CAPABLE
CMake: make CALS driver PLUGIN_CAPABLE
CMake: netCDF: fix MEM and UFFD support
CMake: Cloudant driver can't be plugin due to dep to CouchDB driver
CMake: LVBAG: add GEOS dependency
CMake: pass SQLITE_HAS_COLUMN_METADATA to SQLite and GPKG drivers
CMake: make WCS and HTTP driver separately buildable
CMake: make WMS driver plugin capable
CMake: fix building EXR driver as plugin
CMake: make Java bindings so/dll link against libgdal
CMake: add -Wl,--no-undefined to detect issues with drivers built as plugins
CMake: fix build against Poppler 2021.08
CMake: disable bindings when doing a static build
CMake: robustify use of -Wl,--no-undefined which doesn't work on Mac
CMake: doc: fix CMAKE_INSTALL_PREFIX vs CMAKE_PREFIX_PATH
CMake: make name of 'gdal' target configurable
CMake: change name of library target to GDAL upper case, so that GDAL::GDAL can be imported conforming to existing FindGDAL.cmake distributed with cmake
CMake: make GDAL usable as add_subdirectory() of another project
CMake: FindFreeXL.cmake: look for freexl_i first, for Conda shared library
CMake: make DAP disableable
CMake: tune find and use of PDFium
CMake: make use of libxml2 optional
CMake: make Jasper disableable
CMake: CI: reenable msys and mac builds
CMake: make MySQL disableable
CMake: CI: disable mysql for msys2-mingw
CI: workaround a regression / change of behaviour in JXL
CMake: CI: disable python bindings for msys2-mingw
typo fixes
port/CMakeLists.txt: use correct BUILD_SHARED_LIBS name
scripts/fix_typos.sh: adapt for tree reorganization
typo fixes
CMake: for Windows multi-generator, only generate Python bindings in Release mode
CMake: cleanup setup.py.in
VICAR: fix harmless unsigned-integer-overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40377
VFK/Selafin: avoid issues with GPKG driver when those drivers are built as plugins
apps/GNUmakefile: generate test_ogrsf by default (but not installed)
Export VSI_TIFFOpen() to build the INGR driver as plugin
CMake: make sure that gcore/ if built with OGR_ENABLE_SQLITE
Doc: fix doc building
gcore/CMakeLists.txt: make sure OGR_ENABLE_SQLITE is defined
CMake: gcore/CMakeLists.txt: use correct variable for Xerces
CMake: python bindings: use parallel build
README.md: update links [ci skip]
doc/.azure-pipelines.yml: fix path
CMake: update resource file list to install
CMake: fixes for gdal.pc and gdal-config generation
GML/NAS: avoid false positive -Wnull-dereference -O2 warnings
Warp kernel: avoid writing outside allocated buffer if more threads than needed are allocated
warp VRT: detect inconsistent block size between dataset and bands (fixes #4714)
warp VRT: do not serialize block size at band level, since already serialized at dataset level (refs #4714)
Warped VRT: advertize INTERLEAVE=PIXEL for faster processing of multiband
VRTWarpedDataset::ProcessBlock(): fix issue in the unlikely situation where a block would be > 4 GB
GDALAutoCreateWarpedVRTEx(): avoid potential crash in case of error
autotest: typo fix
gdal_pansharpen.py: display usage if not enough filenames provided
BMP: harden identify checks to avoid misidentification of other datasets (fixes #4713)
CMake: Python: do not regnerate docs/.i file to avoid issues with make clean
CMake: Python: remove obsolete tests on numpy version
CMake: tweaks so that python 3.6 is detected out of the box on ubuntu 18.04
CMake: detect and use -Wzero-as-null-pointer-constant
CMake: fix case of tiledb target, so it works properly as plugin
HOWTO-RELEASE: fixes related to new tree organization [ci skip]
CMake: FindOpenJPEG: add include directories for newer versions
CMake: fix running unit tests under Windows
CMake: set HAVE_STD_IS_NAN in cpl_config.h
autotest/cpp/testclosedondestroydm.cpp: do not error out on JPEG2000 driver
Shorten DO_NOT_DEFINE_GDAL_RELEASE_DATE_AND_GDAL_RELEASE_NAME macro name to avoid MSVC warning
scripts/setdevenv.sh: adapt it for CMake builds
CONTRIBUTING.md: updates for new tree org + CMake hints
swig/python/gdal-utils/scripts: make scripts executable
CMake: fix to avoid cmake to be reinvoked everytime
gcore/gdalsse_priv.h: fix warning about cast
CMake: extent/fix autotest/cpp tests
Doc: netcdf: fix default value for GDAL_NETCDF_IGNORE_XY_AXIS_NAME_CHECKS (refs #4732)
CMake: fix MSVC build
HDF5: fix issue when netCDF and/or HDF5 drivers built as plugins with multidim datasets
fuzzers/: make gdal_fuzzer and ogr_fuzzer build on Windows
CMake: add ctest'ing
fuzzers/build.sh: adapt path for new tree org [ci skip]
CMake: do not include some SSE/AVX tests in ctest list
CMake: fix AVX detection on Windows and other SSE/AVX fixes
Doc: advertize 3.3.3 release
CMake: add -relativeimport when regenerating Python bindings
Solaris compilation fix (master only)
CMake: formatting fix
CMake: fix SQLITE_HAS_COLUMN_METADATA detection, and use of PCRE lib
CMake: remove unneeded target_include_directories() that are got through gdal_target_link_libraries()
CMake: add missing dependency of ogrsf_frmts to generate_gdal_version_h
m4/acinclude.m4: fix detection of fseeko/ftello on netBSD
port/cpl_conv.cpp: fix compilation problem on netBSD
CMake: fix setting of ICONV_CPP_CONST in cpl_config.h
test_osr_ct.cpp: fix wrong use of abs() (spotted on netBSD)
gdal_unit_test: fix failure spotted on netBSD on test_cpl<4>
CMake: check that SQLite3 has the RTree extension
CMake: rename SQLite_HAS_xxx to SQLite3_HAS_xxx for consistency
netCDF: limit SetFromUserInput() use to non file input
CMake: fix dependency to Blosc in Zarr driver
Move tif_float.c/h from GTiff driver to port/, to be able to build Zarr driver as plugin
Export symbols for Zarr driver as plugin
CMake: allow Zarr driver to be built as plugin
CMake: build as many drivers as possible as plugins in Windows Conda build
CMake: fix autotest targets with Ubuntu 18.04
CMake: fix i386 builds
autotest/cpp: do not use assert()
CMake: exclude testepsg from default build
CMake: fix test-fuzzer-* on Windows
TileDB: fix crash when creating array from subdatasets fails
scripts/cppcheck.sh: update for cppcheck 2.7
contour.cpp: remove unneeded nullptr check
TileDB: fix handling of relative paths on Windows
Embeded Python: fixes to load Conda and mingw64 python on Windows
Embeded python: list Python 3.10
netcdf.py: Windows fix
CPLRecode(): fix recoding between UTF-8 and CP_ACP/CP_OEMCP on Windows build that have iconv support
netcdf_multidim.py: Windows fix
netCDF: disable filename recoding to ANSI on Windows for netCDF >= 4.8
CMake: do not build HTTP driver as plugin
pds4.py: make it Windows robust
ogr_gmlas.py: Windows fix
Python bindings: for Windows and Python >= 3.8, automatically add path with [lib]gdal*.dll with os.add_dll_directory()
CMake: reformat
TileDB: fix memory leak (introduced in recent commit)
test_gdalsrsinfo.py: fix when CPL_ERROR_SEPARATOR=; is set [ci skip]
tiff_write.py: adapt checksum for Windows Conda builds [ci skip]
CMake: autotest: fix use of bad variable
Rename NEWS to NEWS.md [ci skip]
mkgdaldist.sh: remove autom4te.cache dir from archive (fixes #4748) [ci skip]
GDALmake.opt.in: update LIBGDAL_CURRENT to the one of the 3.4 branch
CMake: CI: enable all ctest for Linux and Windows conda builds
CMake: fix driver enabling conditions
CMake: JPEG must be defined before GTiff
CMake: Windows CI: set git core.autocrlf to false
CMake: set LIBDEFLATE_SUPPORT for internal libtiff when appropriate
GTiff: explicitly enable strip choping
test_gdal_viewshed.py: tolerate more error messages (needed for Conda cmake builds)
CMake: CI: skip test_ogr_gmlas_huge_processing_time() on Windows Conda builds
MITAB: allow reading MID/MIF files with lines up to 1 million bytes (fixes #3943)
cpl_virtualmem.cpp: add comment (fixes #811) [ci skip]
ogr_carto.py: skip test_ogrsf test on CI (fixes CMake Conda windows failures)
OGRSpatialReference::GetName(): workaround a PROJ 8.2.0 bug for BoundCRS
gdaladdo -clean: remove overviews of mask (fixes #1047)
ODS: avoid huge memory allocations on files abusing repeated cells. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40568
CMake: improve install of Python bindings, and add doc
CMake: set SKIP_BUILD_RPATH YES property on plugins. Helps for Linux builds
CMake: remove useless target properties on binaries
CMake: fixes for MacOSX framework builds
DGNv8: fix build issue due to FlushCache() changes
CMake: set rpath to installed plugins on MacOSX frameworks (fixes #4761)
CMake: add support for building the MSG driver
GTiff: JXL codec: use non-deprecated method to set encoding effort
Unix builds: move generated headers to a generated_headers subdirectory
configure.ac: fix detection of OpenEXR >= 3 (fixes #4766)
CMake: detect OpenEXR >= 3
CMake: add JXL support
CMake: add support for SDK 5.5
ECW: fix test failures with ECW 5.5
Doc: precision for ECW [ci skip]
CMake: fix MrSID support
Fix OSSFuzz build [ci skip]
CMake: fix KEA support
HDF5 and KEA: remove warnings with MSVC
Disable/fix various compiler warnings on CMake Conda Windows builds
MSG driver: fix/workaround MSVC warnings
CMake: MSG: set -DWIN32
CI: turn warning-as-errors on CMake Conda Windows builds
CMake: fix warnings on Mac builds and turn on warning-as-errors
Doc: advertize 3.4.0
CI: disable building C# samples on Windows due to #4792
Doc: download.rst: fix link
FileGDB: add a WKID (undocumented for now) layer creation option
GDALDriver::QuietDelete(): take into account papszAllowedDrivers argument
GDALDataset::MarkSuppressOnClose(): delete auxiliary files (fixes #4791)
CMake: fix typo in FindSQLite3.cmake (fixes #4795)
CMake: FindECW.cmake: add comment (fixes #4796)
Doc: fix links to no longer existing gdal/ subdirectory (fixes #4798)
GTiff: only emit warnings on libgeotiff PROJ errors (fixes #4801)
gdalinfo --build: report PROJ version of build and runtime
Doc: better point to RPC transformer options
CMake: fix detection of SQLite3 extra symbols when linking with a SQLite3 static lib (fixes #4811)
gdal_polygonize.py: remove use of Unicode double quote characters
Docker: fix --release builds
CMake: implement Kakadu library finding
Zarr: be robust to duplicated array and group names in NCZarr metadata, which could lead to performance issues. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40949
CMake: SQLite3: accept variables in upper case
ogrsqlite3ext.h: avoid 'redefinition of typedef 'sqlite3_api_routines' is a C11 feature' warning
CMake: force C99
FileGDB: avoid warning with MSVC
CMake: add FileGDB support
Add support for PCRE2 (to replace deprecated PCRE) (fixes #4822)
CMake: add PCRE2 support (refs #4822)
OGRSimpleCurve::segmentize(): avoid too many memory reallocation (refs #4826)
gdalwarp -crop_to_cutline: relax epsilon to avoid extremely long loop (fixes #4826)
CMake: add support for LURATECH
GRIB: fix thread-safetey of errSprintf() (#4830)
Oracle: fix CMake support and fix compiler warnings (#4812)
/vsis3/: allow setting AWS_PROFILE to a profile that uses IAM role assumption
CMake: fix build issue when re-runing cmake with PCRE present but missing PCRE2 (fixes #4845)
GPKG: fix nullptr dereference on corrupted databases with sqlite >= 3.35
CMake: link wsetargv.obj to GDAL programs with MSVC (fixes #4855)
CI conda: fix path to GDAL sources
Revert "Github workflows: disable Conda builds in pull requests (refs #4599)"
github/workflows/conda.yml: force macos-10.15 to maybe fix build issue
CPLRecodeIconv(): avoid potential unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41201
COG: only create RGB JPEG with mask if 4-band is alpha (fixes #4853)
GTiff: avoid warning with >= 5 band and JPEG compression
GDALPamRasterBand::CloneInfo(): deal correctly with NaN nodata to avoid generating useless .aux.xml file (fixes #4847)
GRIB: avoid read heap buffer overflow due to inappropriate split-and-swap on dataset with weird georeferencing. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41260 (3.4.0 regression)
CMake: for default Python_FIND_STRATEGY=LOCATION, requires Python 3.6 at least (avoid picking Python 2.7 on Mac)
CMake: avoid warning messages for rdb and TileDB
CMake: FindKDU.cmake: fix for CMake < 3.13
ogrsqliteregexp.cpp: fix control flow issue in PCRE2 code. Fixes CID 1466404
ZIP support: avoid warnings on MacOS
SQLite: workaround MacOS system SQLite non-default settings that cause issues with WAL and AlterFieldDefn() when patching CREATE TABLE DDL
VFK: avoid crash when a SQLite3 statement fails
ogr_virtualogr.py: skip when SQLite built with OMIT_LOAD_EXTENSION
ogr_vdv.py: workaround test failure with MacOS system SQLite
ogr_vfk.py: workaround test failure with MacOS system SQLite
CMake: avoid issues with Mono framework on MacOSX
CMake CI: enable tests on Mac builds
TileDB: avoid warnings about deprecated functions
TileDB: avoid clang warning about parenthesis
cpl_compressor.cpp: avoid warning when including lzma.h of Conda with CLang
cpl_xml_validate.cpp: avoid warning with libxml2 of Conda with CLang
port: avoid warnings with zlib of Conda with CLang
SWIG: add missing CPLES_SQLI constant (fixes #4878)
FindGeoTIFF.cmake: locate release and debug libraries (fixes #4854)
FindPROJ.cmake: locate release and debug libraries (fixes #4854)
OpenFileGDB: correctly parse raster fields of type == 2 which are inlined binary content (fixes #4881)
GTIFFWriteDirectory(): remove useless paramater
External overviews: automatically turn PLANARCONFIG_CONTIG for WebP overviews
GTiff: LERC overview related improvements
OGR_CT: add a OGR_CT_PREFER_OFFICIAL_SRS_DEF config option
/vsiaz/: read credentials from ~/.azure/config as an additional fallback method
gdal_sieve.py: fix exception when source dataset has no nodata value (3.4.0 regression) (fixes #4899)
CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe
ossfuzz build: build GDAL against external libjpeg since newer poppler requires jpeg_mem_src() which is new to libjpeg v8 and thus absent from our internal libjpeg v6b
ElasticSearch: do not try to open reserved .geoip_databases to avoid error message with recent Elastic versions
Internal libtiff: resync with upstream
Internal libgeotiff: resync with upstream
Shape: better deal with ETRS89 based CRS with TOWGS84[0,0,0,0,0,0,0]
GDALDatasetCopyWholeRaster(): clarify INTERLEAVE option meaning and raises warning if invalid value is specified (fixes #4909)
WFS: if COUNT is present in WFS >= 2, use it as the page size. Also clarify doc
SQLite: add a STRICT=YES layer creation option to create tables as SQLite >= 3.37 STRICT tables
PDS4: write conformant Equirectangular when input raster is a geographic CRS
Zarr: hopefully fixes 'runtime error: reference binding to null pointer of type 'unsigned char''. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41517
Zarr: fix cppcheck warning added in previous commit
ODS: avoid crashing 'floating-point exception' when evaluating -2147483648 % -1. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41541
GTiff: fix performance issue when reading transfer functions (fixes #4923)
Internal libtiff: resync with upstream
/vsiaz/: allow authorization through an access token specified with the AZURE_STORAGE_ACCESS_TOKEN config option (to be used with AZURE_STORAGE_ACCOUNT)
Move multi-line CSV parsing function from OGR driver to cpl_csv.h
MITAB: fix parsing .mid files with newline character in string field (fixes #4789)
GRIB: avoid read heap buffer overflow due to inappropriate split-and-swap on dataset with weird georeferencing. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41637 (3.4.0 regression)
GTiff: propage SPARSE_OK to overviews (refs #4932)
Fix -Wnull-dereference warnings with gcc in -O3 -DNDEBUG
GTiff: avoid huge memory allocation when generating overviews on large single-band 1-bit tiled files (fixes #4932)
SQLite: fix memory leak recently introduced (master only). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41671
tif_jxl.c: adapt for latest jxl master API changes
.github/workflows/ubuntu_20.04/build-deps.sh: force rebuild of dep
Revert "CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe"
TopoJSON: fix duplicate 'id' field, and other potential issues when reading fiels (3.4.0 regression)
Internal libtiff: fix issue with rewrite-in-place logic of https://gitlab.com/libtiff/libtiff/-/issues/309
PCIDSK: fix write heap-buffer-overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41993
frmts/georaster/CMakeLists.txt: use correct variable to decide to link against external libz
configure: change default of RENAME_INTERNAL_LIBTIFF/LIBGEOTIFF/SHAPELIB_SYMBOLS to yes
Unix/CMake build: when building against internal libjpeg, prefix by default libjpeg symbols with gdal_ (fixes #4948)
CMake: add support for Teigha/ODA libraries
Internal libtiff: resync with upstream to get https://gitlab.com/libtiff/libtiff/-/merge_requests/277
GDALDefaultCSVFilename(): remove remains of GeoTIFF EPSG CSV files era
Typo fixes
GRIB: fix writing negative longitude of natural origin for Transverse Mercator, and fix reading it for TMerc, LCC, ACEA and LAEA
Internal libtiff: LZW codec: fix support for strips/tiles > 2 GB on Windows
tif_lzw.c: fix warnings of previous commit
tif_lzw.c: other warning fixes
PG: error out if non UTF-8 content is transmitted in COPY mode, when client_encoding=UTF8 (refs #4751)
S57: enable recoding to UTF-8 by default (RECODE_BY_DSSI=YES open option) (fixes #4751)
tiff_write.py: typo fix [ci skip]
COG: fix potential generation failure when main imagery has overview and mask none
Add GDALPamDataset::DeleteGeoTransform() (refs #4877)
GTiff: make SetGCPs(), SetGeoTransform(), SetSpatialRef(), SetNoDataValue(), SetMetadata[Item]() write to PAM .aux.xml on read only files (fixes #4877)
test_gdal_edit.py: test -ro switch (refs #4877)
Doc: fix broken links to /vsi file systems (fixes #4967)
Internal libtiff: have TIFFJPEGIsFullStripRequired properly renamed for RENAME_INTERNAL_LIBTIFF_SYMBOLS builds and simplify tif_jpeg build
frmts/gtiff/libtiff/CMakeLists.txt: adapt from previous commit
fuzzers/build.sh: fix PROJ linking
GTiff: add support for reading/writing color table from/into PAM .aux.xml (fixes #4897)
PCIDSK SDK: fix harmless unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42120
Remove obscure test program xmlreformat (that was not built by default)
nmake.opt: fix wrong variable name in example MSODBCSQL_LIB [ci skip]
CMake: fix ODBC on Windows, and detect MSSQL native CLI or ODBC (fixes #4810)
GTiff: do not warn about buggy Sentinel1 geotiff files use a wrong 4326 code for the ellipsoid
CMake: fix internal libtiff build with jpeg12
CMake: fix MSVC Ninja build
FindMSSQL_ODBC.cmake: add Linux support
CMake CI Linux: add msodbcsql17 client support
MSSQL: fix warnings on Linux
MSSQL: avoid GetLayerDefn() to return null, and call it in GetExtent()
OGRMSSQLSpatialTableLayer::GetLayerDefn(): fix potential memory leak of pszFIDColumn
OGRMSSQLSpatialDataSource::DeleteLayer(): remove suspicious test 'strlen(pszTableName) == 0' that causes a read after free
MSSQL: fix issue when inserting strings in bulk copy mode on Linux
MSSQL: make GetNextFeature() end bulk copy mode (to fix autotest failure)
MSSQL: CreateFeature in bulk copy mode: make it use feature geometry SRID when set (to fix autotest failure)
MSSQL: disable bulk mode when a UUID field is found, as this doesn't work currently
ogr_mssqlspatial.py: adapt test for bulk copy mode
MSSQL: do not set field width when reading smallint/int/bigint/float/real columns, and correctly roundtrip smallint/Int16 (fixes #3345)
Metadata readers: remove thread-unsafe use of localtime()
Various cppcheck fixes (mostly always true/false conditions)
cppcheck fix: avoid 'Returning pointer to local variable 'poInvisibleLayer' that will be invalid when returning'
CMake: fix cryptopp support, and some tidying in check_cxx_source_compiles() of other packages
CMake: fix GTA driver
CMake: disable OpenMP detection
CMake: fix Crnlib detection
CMake: remove detection of unused epsilon package
CMake: add support for IDB
fuzzers/gdal_fuzzer.cpp: avoid int64 overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42251
FindIDB.cmake: add advanced options
Revert "Revert "CI Conda: adapt to https://github.com/conda-forge/gdal-feedstock/commit/9570a0ee7139e292e621a5c286cc405629c8d8fe""
CMake: fix FYBA/SOSI support
CMake: add support for MongoDBv3 driver
CMake: fix support for RasterLite2 and Spatialite
CI CMake Ubuntu: add a few missing dependencies
CMake: add support for HEIF
CMake: add GDAL_ENABLE_FRMT_JP2MRSID option
CMake: mark number of options as advanced
CMake: use generic infrastructure for GTA and KEA detection
CMake: use generic infrastructure for PDFium detection
CMake: use generic infrastructure for CFITSIO and JXL detection
add_javadoc.c: robustness fixes
CMake: remove duplicated FindRasterlite2.cmake
swig/java/add_javadoc.c: fix bad use of strcat() that caused crashes on MacOSX
CMake: fix Java bindings
CMake Java: avoid constant regeneration of maven .jar when GDAL_JAVA_GENERATE_JAVADOC=OFF
Internal libtiff: resync with upstream [ci skip]
OGR GMT: allow writing to /vsistdout/ (fixes #4993)
JP2KAK: add support for reading/writing Int32/UInt32 data types
ECW: do not try to open UInt32 JPEG2000 if the SDK is buggy (fixes #3860)
CMake Python: small tuning to avoid useless run of setup.py
PCIDSK SDK: fix harmless unsigned integer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42420
gdal_polygonize.py: make -8 switch work again (fixes #5000)
CMake: add/fix CSharp support
CMake CSharp: remove .config files for Unix
CMake CSharp: add install target
test_gdalinfo.py / test_gdalinfo_py.py: avoid potential conflicts if run in parallel
Warped VRT: fix issue with blocks without sources and alpha band (fixes #4997)
Remove RDA (DigitalGlobe Raster Data Access) driver
gdalwarp: do not emit "Point outside of projection domain" / "tolerance condition error" (fixes #4934)
CMake DWG: build createdgnv8testfile utility
DWG/DGNv8: avoid potential crash with ODA 2022 on Linux
CMake ODA: add a TEIGHA_ACTIVATION_FILE_DIRECTORY variable
Remove most uses of GDALIsInGlobalDestructor()
CMake: fix CSharp build on Windows
Internal libtiff: resync
CMake FindTEIGHA.cmake: update for ODA >= 2021 on Windows
CMake: fix Visual Studio ALL_BUILD solution
swig/python/GNUmakefile: fix/workaround setuptools 60.0 and Debian --install-layout
CMake Python: fix/workaround setuptools 60.0 and Debian --install-layout
generate_quant_table_md5sum.cpp: use uint16_t (fixes #5021)
configure: detect xlocale.h to use LC_NUMERIC_MASK on Mac (fixes #5022)
cmake: detect xlocale.h to use LC_NUMERIC_MASK on Mac (fixes #5022)
USGSDEM: fix reading datasets with 1025 byte records ending with linefeed (fixes #5007)
autoconf/nmake: Add support for external libqhull_r (refs #4040)
CMake: add support for external libqhull_r (refs #4040)
gdal2tiles: XML escape input filename (fixes #5032)
CMake: fix build with cmake < 3.12
CMake: cleanup w.r.t internal libqhull
delaunay.c: fix warning with MSVC with libqhull_r
Internal libqhull: update to qhull_r 2020.2 (8.0.2) (fixes #4040)
internal_libqhull/poly_r.c: avoid int overflow on 32bit (re-apply 4224afd669eebbf657afe90014436bdc83fd0f34)
Drop support for non-reentrant external libqhull
GDALTriangulationCreateDelaunay(): allow redirecting error stream of GDALTriangulationCreateDelaunay() to temporary file to avoid issues with ctest with MSVC and external libqhull
CMake Windows Conda: use external qhull
CMake: reformat gdal.cmake and CheckDependentLibraries.cmake
CMake: make Armadillo link to LAPACK if needed (Windows Conda builds)
openexr_headers.h: avoid warnings with MSVC
CMake: fix build on Linux with Conda, related to iconv
Avoid warnings related to libxml2
Avoid warnings related to curl
Avoid warnings related to libgeotiff
CPLZLibInflate(): workaround issue with /opt/intel/oneapi/intelpython/latest/lib/libz.so.1 from intel/oneapi-basekit Docker image
CMake: disable -ffinite-math-only with Intel's icx compiler
Internal libqhull: silence Coverity Scan warnings
Internal libtiff: resync with upstream
static'ify method
Remove redundant nullptr checks
CITATION: update year
Tag another method as static
cppcheck.sh: use --library=gnu to avoid warning with anonymous mmap() and fd==-1
autotest/gcore/pixfun.py: attempt at fixing failure on CI build-linux-ubuntu-focal
gdal2tiles: remove/fix broken links
Docker: update Alpine based images to 3.15 [ci skip]
internal libqhull: fix / silence Coverity warnings
gdal2tiles: lower chuncksize value to allow parallel processing when generating ~ 100 tiles
docker/ubuntu-full/Dockerfile: fix for building libjxl [ci skip]
Doc: advertize 3.4.1 release
metaname.cpp: white space fixes
degrib: partial resynchroinzation with degrib 2.25 (degrib-20200921)
degrib: fix cppcheck warning
cpl_csv.h: add a CSVRewind() function
GRIB: mode degrib hard-coded tables to .csv files in the data/ directory
GRIB: merge content from WMO tables at https://github.com/wmo-im/GRIB2 with DEGRIB ones
GRIB: fetch WMO 4.2 subtables
Elasticsearch: add a AGGREGATION open option (#4962)
configure.ac: add a --with-qhull-pkgname=qhull_r/qhullstatic_r option to select with qhull package to use
CMake: add a QHULL_PACKAGE_NAME=qhull_r/qhullstatic_r option to select with qhull package to use
GRIB: use tagged version of the wmo-im/GRIB2 repository
Doc: fix warnings with Doxygen 1.9.3
configure.ac: remove useless QHULL_INCLUDE_SUBDIR_IS_LIBQHULL
ogr2ogr: propagate error in final CommitTransaction() to status code of the utility (fixes #5054)
PG: propagate errors in defered EndCopy() call in CommitTransaction() (fixes #5054)
Elastic: support GetLayerByName() with multiple layers and/or wildcard (fixes #4903)
HDF5: support libhdf5 1.13.0 (fixes #5061)
frmts/pdf/GNUmakefile: force c++17 with Poppler > 21 (fixes #5071)
frmts/pdf/CMakeLists.txt: force c++17 with Poppler > 21 (fixes #5071)
ogr2ogr / PG driver: make ogr2ogr -lco GEOM_TYPE=geography work when the source layer has a named geometry column, and also reject creating geography columns with a SRS != EPSG:4326 (fixes #5069)
GPKG/SQLite: deal explicitly with CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES for /vsi network file systems (fixes #5031)
Python bindings: replace unsafe use of tempfile.mktemp() by mkstemp() (fixes https://huntr.dev/bounties/59e93871-266a-40be-9aad-4576d3dc48f2/)
PG: support other geographic SRS than EPSG:4326 for geographic type (fixes #5075)
gdal2tiles: fix issue with multiprocessing and the gdal2tiles launcher script on Windows and Python >= 3.8 (fixes #4951)
autotest/pymod/test_py_scripts.py: run wrapper scripts of swig/python/gdal-utils/scripts instead of swig/python/gdal-utils/osgeo_utils
GRIB: avoid harmless unsigned integer overflow. master only. fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43479
docker/README.md: more details about grids [ci skip]
doc/source/build_hints.rst: fix copy&paste error
CMake python: move .in files to swig/python
CMake: Python: fix setup.py to work again in standalone mode
Internal libtiff: resync with upstream
CMake: enhance build_hints.rst and various fixes/enhancements
cmake: remove tabulations in .cmake files
CMake: declare option GDAL_USE_PUBLICDECOMPWT
CMake: add support for external LERC library (fixes #5088)
cmake: FindSPATIALITE: fix for Conda windows
cmake: FindMySQL: fix for Conda windows
CI: enable Spatialite on Conda CMake windows builds
msys2 mingw64 build: fix warning about BLOB being defined in a system header
msys2 mingw64 build: fix warning about type mismatch in CPLDebug() and unused argument
msys2 mingw64 build: use ll formatter instead of I64
msys2 mingw64 build: fix weird warning about string overflow
msys2 mingw64 build: avoid -Wnull-dereference warnings
msys2 mingw64 build: avoid warning about invalid reinterpret_cast<>
msys2 mingw64 build: avoid warnings in cpl_vsil_crypt.cpp and cpl_sha256.cpp
msys2 mingw64 build: attempt fixing warning about CPL_FRMT_GIB in ogrosmlayer.cpp
msys2 mingw64 build: fix warning about CPL_FRMT_GIB in NAS driver
CI build-windows-msys2-mingw: turn warnings as errors
AAIGRID: add support for 'null' as can be generated by D12 software (fixes #5095)
GTiff: fix DISCARD_LSB with nodata value (fixes #5097)
Internal libtiff: resync with upstream
Internal libjson: update to 0.15.0
Internal libjson: GDAL specific changes
Internal libjson: compiler warning fixes
Add .zenodo.json file (#5101)
Add FUNDING.yml
.github/FUNDING.yml: add link to numfocus donate to GDAL [ci skip]
OGRSpatialReference::importFromEPSG(): document OSR_USE_NON_DEPRECATED=NO configuration option (refs #5106)
gdalsrsinfo: emit message when replacement of deprecated CRS occurs, with hint to set OSR_USE_NON_DEPRECATED=NO configuration option to avoid that (fixes #5106)
cmake: check linux/fs.h presence
CMake: fix MACOSX_FRAMEWORK setting
CMake: remove unused options, mark some as advanced
CMake: enable -ftrapv for debug builds
CMake: detect __builtin_bswap32 / __builtin_bswap64
CMake: add build support for dummy NULL driver
CMake: add support for libhdfs (for /vsihdfs)
CMake: add missing detection of a few functions
fuzzers/build.sh: build Poppler with sanitize flags to avoid https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43668 [ci skip]
CMake: add GDAL_AUTOLOAD_PLUGINS option
CMake: enable -DDEBUG for debug builds
Add a prefix/lib/gdalplugins/drivers.ini file to control registration order
GTiff: RoundValueDiscardLsbUnsigned(): fix potential unsigned overflow (CID 1469012)
Internal libjson: add assertion to avoid CID 1469011
Internal libjson: fix printf formatter (CID 1448587)
RFC 84: modify planned schedule according to proposal of https://lists.osgeo.org/pipermail/gdal-dev/2022-January/055253.html
CI ASAN: port to Ubuntu 20.04 (fixes #5124)
OGREnvelope: add == and != comparison operators
WMTS: disable clipping with TileMatrixSetLimits by default when using layer extent and add open options to control that behavior (fixes #4461)
WMTS: document open options (fixes #4471)
VSISync() onto /vsis3/: allow x-amz- headers to be specified for object creation
/vsis3/: ignore object class DEEP_ARCHIVE in addition to GLACIER, and add CPL_VSIL_CURL_IGNORE_STORAGE_CLASSES to configure which object classes should be ignored
netCDF: lint: better variable name
gdalbuildvrt: change logic to check homogenous number of bands (refs #5136)
Python bindings and utilities-as-lib: propagate warnings to custom error handler when UseExceptions is on (fixes #5136)
gdalbuildvrt: add a -strict/-non_strict flag, and in strict mode consider non-existing datasets as a failure (fixes #4755)
Add DOI to README.md and CITATION [ci skip]
CITATION: add doi to BibTex entry [ci skip]
Doc: add DOI to about page
byndataset.cpp: whitespace fixing [ci skip]
BYN: remove validation of nTideSys and nPtType fields
netCDF: add WRITE_GDAL_VERSION and WRITE_GDAL_HISTORY creation option
Updates for IJG libjpeg-9e
/vsigs/, /vsiaz/, /vsiadls/: allow x-goog- / x-ms-/ headers to be specified for object creation
Replace reference to MIT/X or X/MIT to MIT (fixes #5154)
CMake: use Poppler_ for variable names
Update swig/csharp/dll_template.csproj
CMake CSharp: fix dependency rules so that a second 'make' without any change doesn't result in .dll and sample files to be regenerated
Add RFC 85: Policy regarding substantial code additions (#5128)
CI: fix crash on Windows Conda CMake builds
/vsis3/: make sure file properties of /vsis3_streaming/foo are invalidated when /vsis3/foo ones are
CI: disable build-windows-conda-csharp-only job
OGR MEM driver: set GDAL_DCAP_FIELD_DOMAINS and GDAL_DMD_CREATION_FIELD_DOMAIN_TYPES metadata items (instead in the raster MEM driver) [master only]
TGA: fix reading images with runs crossing scanlines (fixes #5168)
CMake: MBTiles: fix for external Zlib
Doc: add L3Harris/Harris Geospatial Solutions as bronze sponsor
Add forgotten file
CMake: fix build with libxml2 that may require Iconv
CMake FindHDF4: fix how we look for ws2_32
nitf.py: workaround issue with Conda Windows and libjpeg-9e
gdal.i: remove duplicated GDAL_RMS constant
netCDF: add a VARIABLES_AS_BANDS=YES/NO open option
netCDF: allow update mode of raster datasets
netCDF: implement SetMetadataItem()/SetMetadata()
netCDF: avoid warnings when CreateCopy() a non-georeferenced dataset, and opening a 1x1 non-georeferenced dataset
Update version numbers to 3.5.0/3.5.0dev
HOWTO-RELEASE: add a step to bump version numbers for the -dev version [ci skip]
Driver manager: make sure oMapNameToDrivers.size() == nDrivers when querying a non existing driver
GDALDriverManager::ReorderDrivers(): add assertion
ogr_gpkg.py: remove temporary file
CMake: fix minimum build with internal zlib, libjpeg and libtiff
CMake: swig/python/setup.py.in: use GDAL_LIB_OUTPUT_NAME variable
CI CMake: add a minimum build configuration
GPKG: when adding a 'epoch' column to gpkg_spatial_ref_sys, one must use the 'gpkg_crs_wkt_1_1' extension instead of 'gpkg_crs_wkt'
Travis CI: fix getting get-pip.py for Python 3.6
GPKG: DeleteField(): use ALTER TABLE ... DROP COLUMN if sqlite >= 3.35.5, and run foreign_key_check only if foreign_keys is ON (fixes qgis/qgis#47012)
Move GDALGeoPackageDataset::PragmaCheck() to OGRSQLiteBaseDataSource
SQLite: DeleteField(): use ALTER TABLE ... DROP COLUMN if sqlite >= 3.35.5, and run foreign_key_check only if foreign_keys is ON (refs qgis/qgis#47012)
GPKG: AlterFieldDefn(): use ALTER TABLE ... RENAME COLUMN when only renaming is to be done and sqlite >= 3.26.0, and do not run integrity_check in that situation (fixes qgis/qgis#47012)
SQLite: AlterFieldDefn(): use ALTER TABLE ... RENAME COLUMN when only renaming is to be done and sqlite >= 3.26.0
SQLite/GPKG: avoid CPLDebug() error message when executing ExecuteSQL("PRAGMA table_info(..)")
GPKG: intercept 'ALTER TABLE table RENAME COLUMN src_name TO dst_name' and 'ALTER TABLE table DROP COLUMN col_name'
CMake: implement documented GDAL_USE_TIFF switch, and avoid copy&paste patterns
CheckDependentLibraries.cmake: remove unsued HAVE_XMLPARSER variable
CheckDependentLibraries.cmake: use gdal_check_package() for OpenSSL
CMake: make all dependencies found disableable
CMake: add a GDAL_USE_EXTERNAL_LIBS=OFF option to disable all external libs
build_hints.rst: document missing GDAL_USE_ variables
CMake: rename GDAL_ENABLE_FRMT_ variables to GDAL_ENABLE_DRIVER_ and OGR_ENABLE_ to OGR_ENABLE_DRIVER_
CMake: fix use of undefined variables
CMake: rename GDAL_JPEG12_SUPPORTED to GDAL_USE_LIBJPEG12_INTERNAL
CMake: fixes for cmake/template/gdal_def.h.in
CMake: warn when turbojpeg is used
frmts/netcdf/CMakeLists.txt: cleanup
CMake: allow to build HTTP driver as a plugin
CMake: rename GDAL_USE_LIB{foo}_INTERNAL to GDAL_USE_{foo}_INTERNAL
CMake: fix 'export ordinal too large' error with mingw and default build type
netcdfdataset.cpp: use std::tie()
GTIFPrintDefn(): avoid potential nullptr deref. CID 1469240
plmosaicdataset.cpp: explicitly ignore ret value of VSIMkdir() (CID 1469242)
Internal libjson: complement to fix CID 1448587
ILI IMDReader: fix various potential crashes on invalid output (CID 1469239, 1469241, 1469243, 1469244, 1469245)
CMake: fix build of DDS driver as a plugin
frmts/drivers.ini: fix registration order of GNM drivers
CMake: fix installing drivers.ini in gdalplugins directory of build tree
Delete .DS_Store [ci skip]
cmake_builds.yml: disable ctest on build-windows-conda job
Revert "cmake_builds.yml: disable ctest on build-windows-conda job"
cmake_builds.yml: build-windows-conda: disable HTTP driver (as before 6b05d80f01c86880e5729fb59243dba8a1f48e49)
GPKG: add a NOLOCK=YES option to open a file without any lock (for read-only access) (helps fixing qgis/qgis#23991, but requires QGIS changes as well)
GTiff: fix exposing WEBP_LOSSLESS option
CMake: GTiff: fix exposing WEBP_LOSSLESS option
autotest/pymod/webserver.py: fix to not start a server on a port already occupied on Windows
gdalhttp.py: skip test_http_ssl_verifystatus() on build-windows-conda
Revert "cmake_builds.yml: build-windows-conda: disable HTTP driver (as before 6b05d80f01c86880e5729fb59243dba8a1f48e49)"
CMake: make it use CMake config files in priority to find PROJ
CMake: use gdal_target_link_libraries() to link PROJ
CMake: align signature of gdal_target_link_libraries() on standard target_link_libraries()
CI: ogr_gmlas.py: skip HTTP related tests on build-windows-conda
Remove doc/.azure-pipelines.yml given that we now have a github action for that [ci skip]
CMake: generalize use of gdal_target_link_libraries() to non-driver object libraries
doc_build.yml: fix
MVT writing: using MakeValid() when possible on polygon output...
Add missing const references in argument passing
netcdfdataset.cpp: remove always true conditions
MapInfo .tab writing: correctly detect datum when creating a layer from a WKT2 CRS string (fixes #5217)
FileGDB: workaround a crash involving binary field and CDF datasets
CMake: fix minimal build without JPEG driver
CI: adapt for windows-latest environment which is now windows-2022, and explictly use that version
autotest/cpp: relicence LGPL licensed files to MIT except one exception
CMake: fix 'Invalid escape sequence' warning on Python bindings install on Windows
CI CMake: turn on -Werror=dev
SWIG: add gdal.DataTypeUnion()
README.md: update badge for docs [ci skip]
netCDF: add a IGNORE_XY_AXIS_NAME_CHECKS=YES open option (refs qgis/QGIS#47158)
gdal_calc.py: support wildcard for filenames, and use 3D arrays instead of list for multiple filenames per alpha
Warped VRT opening: do not open the source dataset with GDAL_OF_SHARED
GDALAutoCreateWarpedVRT(): clarify source dataset ownership with example
bash-completion: fixes and update gdal-bash-completion.sh
Rename cmake/helpers/GdalSetTestEnv.cmake to GdalSetRuntimeEnv.cmake and use it in autotest/CMakeLists.txt
scripts/setdevenv.sh: add /home/even/gdal/gdal/swig/python/gdal-utils to PYTHONPATH for autoconf builds
CMake: add a 'completion' target and install bash completion scripts (fixes #5232)
CMake: install python: fix DESTDIR support
CMake: make URL of PublicDecompWT SDK configurable and use my github clone for CI since eumetsat gitlab is down
/vsitar/: fix reading .tar.gz files when the size of the uncompressed .tar file is a multiple of 65536 bytes (fixes #5225)
GDALAutoCreateWarpedVRT doc: better code example formatting
VRTWarp: fix serialization of relativeToVRT=1 when mix of relative and absolute paths for source and VRT (refs #4983)
VRT (non warped): allow setting VRT description after CreateCopy('', src_ds)
VRT (non warped): fix serialization of relativeToVRT=1 when mix of relative and absolute paths for source and VRT
CPLGetCurrentDir(): use _wgetcwd() on Windows to get a UTF-8 filename
CMake: remove getcwd related settings in MSVC case
configure.cmake: cleanups in MSVC case
Rasterization of polygons: avoid underflow/overflow of output data type
Rasterize: use rounding to integers of floating-point value, and handle +/- inf for floating point rasters
Delete .github/LICENSE.TXT [ci skip]
CMake: internal libtiff: fix build with external libjpeg and internal libjpeg12
Python bindings: commit regenerated files [ci skip]
tif_jxl: bring back support for v0.6.1 since there isn't yet any libjxl release with JxlEncoderFrameSettingsSetOption()
JP2KAK: fix build on Windows (fix ed327928c89841a81023ad67165f6c428ead7d50) [ci skip]
frmts/gtiff/libtiff/CMakeLists.txt: fix build without JPEG
GeoRaster driver: fix build without JPEG
CMake: GeoRaster: fix build without JPEG
CMake: tidy linking of Windows system libraries
ogrlineref.cpp: fix cppcheck warning
CMake: apps/CMakeLists.txt: avoid useless liking of external libraries in binaries
Internal libpng: fix memleak on corrupted file. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44486
GRIB: avoid harmless unsigned integer overflow. master only. fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43652
Build remove message about experimental status of CMake system, and add messages about deprecation of autoconf&nmake (fixes #5171)
Remove deprecated testepsg utility (fixes #3993)
/vsigs/: fix upload of files > 4 MB in HTTP 1.1 (fixes #5267)
GTiff: SRS reader: avoid floating-point division by zero on invalid file. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44502
CMake: fix build with BUILD_APPS=OFF (fixes #5275)
CMake: make get_dep_libs() work with several library names passed to a _LIBRARY variable (fixes #5272), and also fix variable inversion for INTERFACE_LIBRARY
drivers.ini: add NUMPY driver [ci skip]
gdal_calc.py: raise error when overwriting behavior is apparently wished but not specified (fixes #5270)
CMake: fix building without GNM (fixes #815)
CheckDependentLibraries.cmake: fix CMAKE_MSVC_RUNTIME_LIBRARY value for Armadillo detection
FindHDF4.cmake: fix version fetching, which caused a weird 'CMake Error: -D must be followed with VAR=VALUE' later in FindNetCDF...
FindNetCDF.cmake: add needed inclusion
CMake: use the policy_max option of cmake_minimum_required() to avoid turning on manually policies
JP2OpenJPEG: add a STRICT=YES/NO open option to allow decoding some broken files (required OpenJPEG 2.5)
OGRSpatialReference::SetFromUserInput(): make it work with 'IAU:XXXX' and 'IAU:2015:XXXX' (when using PROJ >= 8.2)
autotest/gcore/vsis3.py: remove temporary file [ci skip]
ECW: fix non-nearest upsampling on multi-band datasets (fixes #5288)
CMake: do not use PATH when finding a package config file
GRIB: degrib: fix use of unitialized memory on some datasets (refs #5290)
GRIB: fix write heap-buffer-overflow when using 'split and swap column' mode when the split column is not equal to width / 2 (fixes #5290)
netCDF: recognize x/y axis from GMT generated files as geospatial axis (fixes #5291, fixes qgis/QGIS#47158, fixes qgis/QGIS#45704)
Revert "CMake: do not use PATH when finding a package config file"
Internal libtiff: resync with upstream
GTiff: early checks for PREDICTOR settings, and update internal libtiff to support PREDICTOR=2 for 64-bit samples (fixes rasterio/rasterio#2384)
Python bindings: remove all traces of distutils (refs https://github.com/OSGeo/gdal/pull/5296#issuecomment-1037460842)
gdal.cmake: remove 'set(CMAKE_MACOSX_RPATH ON)' as this is the default behavior with CMake 3.0
VRT: Log10PixelFuncHelper(): micro optimization in complex case
Remove outdated TODO comment [ci skip]
Fix doc building with breathe 4.33.0
Unix/CMake build: prefix internal libpng symbols with gdal_ (fixes #5303)
CMake: add gdal_add_vendored_lib() to ease using internal libs, with potential renames, and use it
CMake: rename by default internal zlib symbols
CMake: mark RENAME_ options as advanced
CMake: rename by default internal giflib symbols
OGRSpatialReference::SetFromUserInput(): do not emit error message on unrecognized string when ALLOW_FILE_ACCESS=NO (fixes Toblerity/Fiona#1063)
gdalwarp: avoid useless use of CHECK_WITH_INVERT_PROJ when computed bounds are at the 'edges' of the geographic domain
warper: better guess output bounds when warping from a rotated pole projection that include poles
Doc: fix python_samples.rst (fixes #5205)
Doc: fix previous commit
Overview generation: fix Cubic resampling on boundaries between valid and transparent areas (fixes #5016)
CI: disable build-windows-msys2-mingw (refs #5317)
OGR SQLite dialect: avoid being confused by an attribute and geometry field of same name
CSV layer: when the .csvt indicates WKT and in default KEEP_GEOM_COLUMNS=YES mode, prefix the geometry field name with 'geom_'
GeoPackage: add ST_MakeValid() SQL function if not linking against Spatialite or if Spatialite lacks ST_MakeValid()
Shapefile writer: avoid considering rings slightly overlapping as inner-outer rings of others (refs #5315)
CODE_OF_CONDUCT.md: point to website code of conduct [ci skip]
Shapefile writer: fix perf issue when writing multilinestring with lots of parts (fixes #5321)
GTiff: remove limitation to 32,000 bytes when writing the GDAL metadata tag (fixes #4116)
Shapefile writer: consider rings at non-constant Z as outer rings (fixes #5315)
Disable cmake-format
CMake: avoid error when importing broken expat::expat target on Ubuntu 21.10 (fixes #5324)
COG: clarify that ALIGNED_LEVELS effect starts at 2 or more (fixes #5336)
CEOS2 driver: avoid false positive from msys mingw64 about 'array subscript 'double[0]' is partly outside array bounds of 'int[1]' [-Werror=array-bounds]'
MSSQLSpatial: fix warnings and build errors on msys mingw64 with msodbcsql.h
CI: fix build-windows-msys2-mingw builds
Add a OSR_DEFAULT_AXIS_MAPPING_STRATEGY configuration option
autotest: updates to enable it to pass with OSR_DEFAULT_AXIS_MAPPING_STRATEGY=TRADITIONAL_GIS_ORDER
CMake: fix handling of geotiff-config.cmake
COG driver: output exactly square pixels when using -co TILING_SCHEME (fixes #5343)
warp: ComputeSourceWindow(): round source coordinates when very close (refs #5343)
PDF: fix build against Poppler > 22.2
fuzzers/build.sh: build against Poppler > 22.2
warper: avoid using any resampling kernel when doing just a subsetting of the source image (alignement on pixel boundaries) (refs #5345)
libtiff: resync with upstream
CSLTokenizeString2(): make it work with strings > 2 GB
GTiff: code linting regarding compression numeric code <--> string conversions
CMake: add a warning message if a gcc/clang single config build results in a non-optimized build
Docker README.md: update versions [ci skip]
CI: re-enable libxml2 for build-windows-msys2-mingw
gdal_rpc.cpp: fix warning about untrusted allocation size (CID 1475177)
internal libqhull: initialize a variable (CID 1475189)
gdalmdiminfo_lib / gdalmdimtranslate_lib: try to avoid false positive warning about nullptr deref (CID 1475122, 1475115, 1475134)
gdalmdimtranslate_lib: avoid potential nullptr deref (CID 1475176)
GRIB: avoid false positive warning about nullptr deref (CID 1475140)
GRIB: avoid false positive warning about poDS being tainted... (CID 1475169)
geotiff.cpp: avoid warnings about overflow_before_widen (CID 1475105, 1475116, 1475204)
BAG: avoid warnings about overflow_before_widen (CID 1475131, 1475142)
netcdfdataset.cpp: avoid false positive warning about nullptr deref (CID 1475193)
netcdfsgwriterutil.cpp: silence false positive warning about null dereferenced (CID 1475171)
PCIDSK SDK: silence false positive warning about string non null terminated (CID 1475184)
PCIDSK SDK: avoid false positive warning about resource leak (CID 1475126)
PCIDSK SDK: silence false positive warning about string non null terminated (CID 1475183)
PCIDSK SDK: avoid false positive warning about resource leak (CID 1475145)
PCIDSK SDK: avoid false positive warning about uninitialize members (CID 1475173)
PCIDSK SDK: avoid false positive warning about uninitialize members (CID 1475190)
PDF: avoid using a moved object (CID 1475123)
PDF: avoid false positive warning about double free (CID 1475205)
gdalrasterband.cpp: avoid false positive warning about overflow_before_widen (CID 1475108, 1475110, 1475125, 1475132, 1475191)
overview.cpp: avoid false positive warning about overflow_before_widen (CID 1475127, 1475168, 1475202)
GDALRasterBand::IRasterIO(): avoid potential integer overflow (CID 1475186)
RawRasterBand::IRasterIO(): avoid potential integer overflows (CID 1475151, 1475198)
GNMRule: avoid uninitialized members in constructors (CID 1475155, 1475182)
libopencad: avoid false positive warning about uninitialized mat variable (CID 1475187)
CSV: avoid coverity scan warning about untrusted value (CID 1475200)
DXF: avoid potentially uninitialized members in DXFMLEADERLeader (CID 1475164)
FileGDB: avoid warning about unchecked return value (CID 1475162)
GPKG: avoid potential nullptr deref (CID 1475107, 1475124)
MITAB: fix memleak in error code path (CID 1475129)
MVT: avoid false positive warning about nullptr deref (CID 1475137, 1475148)
OpenFileGDB: avoid false positive warning about nullptr deref (CID 1475133, 1475196)
SQLite: avoid potential nullptr deref (CID 1475158, 1475112)
cpl_time.cpp: avoid overflow_before_widen warning (CID 1475180)
Avoid warnings about curl_easy_setopt() return value not being checked
LercLib: fix warning about comparison to uninitialized m_nCols member in copy constructor (CID 1475121)
SQLite: fix crash when doing select load_extension('libgdal.so') from a statically linked sqlite3 console application
CMake: fix error in gdal_generate_config() when CMAKE_C_IMPLICIT_LINK_LIBRARIES is empty
CI: make ctest actually do something with cmake 3.10.2 on Ubuntu configuration
SQLite: add a OGR_SQLITE_LOAD_EXTENSIONS configuration option
gdal2tiles: detect write error when creating tiles, and allow outputing to /vsi filesystems (fixes #3382, fixes #5370)
fuzzers/build.sh: fix ossfuzz build [ci skip]
Python bindings: make sure GetUseExceptions() doesn't clear the error state (fixes #5374)
gdal2tiles: display overview progress bar only if there are overviews (master only)
swig/python/setup.py.in: fix for standalone builds
CI: test gdal-utils package (fixes #5344)
SQLite: fix VirtualShape support with spatialite 5.0.1 or older and sqlite 3.38.0 (unbreaks CI CMake MacOS target)
SQLite: amend code comment [ci skip]
CI: enable ccache on other CMake configs
CMake: fix dependencies of pytest_runner
gdal.cmake: emit warning for Ninja MSVC builds without explicit CMAKE_BUILD_TYPE [ci skip]
CI build-windows-conda: add blosc dependency [ci skip]
Internal libtiff: resync for OSGeo/libgeotiff#66 fix (which doesn't seem to affect GDAL)
CI: fix and enable msys2 tests
CI build-windows-msys2-mingw: use default msys2 shell for running ctest
Typo fixes
GeoJSON: report OFSTJSON subfield type for properties we can't map to a native OGR type, and better handli…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the bug or the crash?
Renaming a column's name of a large gpkg file (~18 GB) using the DB Manager takes about 30 minutes.
Example picture with a different file:
Having a quick look at gpkg spec at https://www.geopackage.org/spec/ makes seem gpkg to be hierarchically structured such that I assume not every column name of each value needs to be rewritten individually which would explain such long processing times. If a gpkg is well structured I wonder about such long processing times. Or is the reason that the file is instantly rewritten entirely at the moment of the edit?
Doing the same rename in a PostgreSQL Database takes about 2 seconds.
Steps to reproduce the issue
Load gpkg file from https://drive.google.com/file/d/1rEBI600wM54Ztm9CbK4-7i6YUKQq1uNH/view?usp=sharing into the DB manager and change one column's name using the "edit column" button.
Example picture with a different file:
After hitting the OK button the DB Manager and QGIS are stalled for half an hour
Versions
QGIS version
3.22.3-Białowieża
QGIS code revision
1628765
Qt version
5.15.2
Python version
3.9.5
GDAL/OGR version
3.4.1
PROJ version
8.2.1
EPSG Registry database version
v10.041 (2021-12-03)
Compiled against GEOS
3.10.0-CAPI-1.16.0
Running against GEOS
3.10.2-CAPI-1.16.0
SQLite version
3.35.2
PDAL version
2.3.0
PostgreSQL client version
13.0
SpatiaLite version
5.0.1
QWT version
6.1.3
QScintilla2 version
2.11.5
OS version
Windows 10 Version 2009
Active Python plugins
GeoCoding
2.18
GroupStats
2.2.5
HCMGIS
21.8.28
mmqgis
2021.9.10
Qgis2threejs
2.6
QuickOSM
2.0.0
quick_map_services
0.19.27
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.5
processing
2.12.99
sagaprovider
2.12.99
Supported QGIS version
New profile
Additional context
No response
The text was updated successfully, but these errors were encountered: