Large Projects: add limitations to the manual #14000

qgib · 2011-06-23T02:02:25Z

Author Name: Jaideep Gupte (Jaideep Gupte)
Original Redmine Issue: 4013
Affected QGIS version: master
Redmine category:documentation_and_help

Hi, I'm on 1.7 (on snow leopard). I'm have trouble opening a project with 206 layers (mix of vector and raster). Only 84 layers will open, the remaining 122 will be flagged as bad layers. Nothing I do in the Handle Bad Layer window makes any difference, all layers are lost. When I individually try and add these layers to the project, they are not recognised (error: "not a valid or recognized data source"). However, each of these layers opens flawlessly in a new project, or a project with less than 84 layers already open.

Is there a maximum limit for layers? Is this related to a cache issue? Any help would be most appreciated. Thanks!

qgib · 2011-06-27T08:18:38Z

Author Name: William Kyngesburye (@kyngchaos)

Same limit here, 84 layers. Using master build 6d788eb from June 4. And I'm creating a new project, not opening an old one.

qgib · 2011-06-30T15:17:42Z

Author Name: William Kyngesburye (@kyngchaos)

From Royce Cline: the limit of the number of open files per process on OS X is 256. I now remember this problem a while back, but figured it was solved somehow. But then, I don't add a lot of shapefiles to a QGIS project, until recently, usually it's postgis connections.

Royce's solution has been to set ulimit in a Terminal and start QGIS from there, but this is not something for the average user, and is generally a bad way (at least the wrong way) to start any application on OS X.

ulimit only affects processes started from a shel (Terminal). OS X apps get their limits from sysctl/launchctl. You can check it with:

launchctl limit

which for me on 10.6 shows 256 soft limit and unlimited hard limit. Huh, if a soft limit only gives the app a warning but lets it continue opening files, maybe there is a problem in how Qt handles it.

The sysctl limits look different:

kern.maxfiles: 12288
kern.maxfilesperproc: 10240

qgib · 2011-08-29T20:36:48Z

Author Name: Paolo Cavallini (@pcav)

category_id was configured as Project Loading/Saving
pull_request_patch_supplied was configured as 0

qgib · 2011-09-24T00:30:15Z

Author Name: Jürgen Fischer (@jef-n)

priority_id was changed from 6 to Normal

qgib · 2011-12-16T13:12:42Z

Author Name: Giovanni Manghi (@gioman)

fixed_version_id was configured as Version 1.7.4

qgib · 2012-04-16T04:28:39Z

Author Name: Paolo Cavallini (@pcav)

version was configured as master
fixed_version_id was changed from Version 1.7.4 to Version 1.8.0
crashes_corrupts_data was configured as 0

qgib · 2012-06-21T05:48:41Z

Author Name: gcarrillo - (gcarrillo -)

Same issue here. In Linux the limit seems to be 1024 by default. I wonder how other software address this problem since it is platform-specific.

qgib · 2012-06-21T05:52:20Z

Author Name: Giovanni Manghi (@gioman)

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

qgib · 2012-06-22T03:18:17Z

Author Name: gcarrillo - (gcarrillo -)

Giovanni Manghi wrote:

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

Giovanni, if you read what I wrote, you'll see that I'm aware it's not a QGIS problem but a platform (OS) one. My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

qgib · 2012-06-23T02:04:22Z

Author Name: Giovanni Manghi (@gioman)

My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

I guess fist we have to know if other GIS packages overcome this limitation. They do? any open source?

qgib · 2012-09-04T10:01:30Z

Author Name: Paolo Cavallini (@pcav)

fixed_version_id was changed from Version 1.8.0 to Version 2.0.0

qgib · 2013-07-14T09:17:15Z

Author Name: Daniel Vaz (Daniel Vaz)

I think that a good solution would be to keep the user informed about OS limitations.

In the GRASS Wiki they report OS limitations to users (see Number of open files limitation): http://grasswiki.osgeo.org/wiki/Large_raster_data_processing

So let's keep the information accessible by users and create/update a wiki page about Know Issues.

Thanks in advance.

qgib · 2013-07-14T09:24:17Z

Author Name: Daniel Vaz (Daniel Vaz)

Closing related Bug #15657.

qgib · 2013-07-20T06:08:28Z

Author Name: Daniel Vaz (Daniel Vaz)

I created a wikipage

[[Known Issues]]

I think there is no more reason to keep this issue open.

Please, if necessary reopen it.

qgib · 2013-07-21T21:41:10Z

Author Name: Paolo Cavallini (@pcav)

Wiki pages are essentially out of reach for most users. Please add these notes to the manual, and to the Help button.

subject was changed from Large Projects to Large Projects: add limitations to the manual
category_id was changed from Project Loading/Saving to Documentation and Help

qgib · 2013-11-03T09:59:37Z

Author Name: Larry Shaffer (Larry Shaffer)

The original issue (as reported for Mac) has been fixed in master branch with commit d8e5c2f

qgib · 2014-06-28T05:40:43Z

Author Name: Jürgen Fischer (@jef-n)

fixed_version_id was changed from Version 2.0.0 to Future Release - Lower Priority

qgib · 2014-06-28T08:24:50Z

Author Name: William Kyngesburye (@kyngchaos)

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

When I added a QgsLogger line to the setrlimit() if block, above the QgsDebugMsg line, to check if the rlimit was changed, it then worked in a release build. I wonder then, since QgsDebugMsg() is a conditional define that does nothing in a release build, then the whole if block gets optimized to nothing since the if block is essentially empty. When I added QgsLogger(), that's a real function, so the if block is then not empty, and it then works. Maybe the setrlimit() part of the if test needs to be put in a variable and the variable tested, so it doesn't get lost in optimization?

A side note: in searching for an example of how to get non-debug messages into the console log so I can check the rlimits in a release build, I noticed that non-debug messages are not getting to console any more. The usual QgsMessageLog::logMessage() that I see in gqisapp.cpp, that showed up in console logs in the past do not now. And I couldn't find any QGIS log file, it doesn't look like there is a default, but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. I had to use QgsLogger::warning() to get anything to show in the console, and that's only when running from a Terminal, directly not with the open command.

qgib · 2014-06-28T09:49:14Z

Author Name: William Kyngesburye (@kyngchaos)

It wasn't debug vs. release build that was the problem. It's starting from Terminal vs. starting as a normal application. I started the debug build from a terminal to see the debug output for the rlimit settings. Then I started my hacked release build to also see the console output. But when starting either debug or hacked release build as a normal app, the rlimits are not changed and I can't load more than 84 shapefiles.

So, this bug is not fixed yet.

qgib · 2014-06-28T09:52:36Z

Author Name: William Kyngesburye (@kyngchaos)

And same with my old v2.2 - opens more than 84 shapefiles when started from Terminal.

qgib · 2014-07-04T09:59:01Z

Author Name: Larry Shaffer (Larry Shaffer)

William Kyngesburye wrote:

... I noticed that non-debug messages are not getting to console any more. ... but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. ...

I think that has been the case since Lion, but definitely for Mt. Lion and Mavericks. Maybe something to do with sandboxing changes.

I use the following in the .app's @Info.plist@ for my local dev builds:

<key>LSEnvironment</key>
<dict>
  ...
  <key>QGIS_LOG_FILE</key>
  <string>/Users/<my_user>/Library/Logs/QGIS/qgis-dev.log</string>
  ...
</dict>

But this may not be reasonable for the stable version, because there is no shell-type expansion for any of those @LSEnvironment@ values. I suppose you could set that to a log file location writable by all users, but would be best in @~/Library/Logs/QGIS/@.

Retesting the file limit issue now.

qgib · 2014-07-04T10:20:57Z

Author Name: Larry Shaffer (Larry Shaffer)

William Kyngesburye wrote:

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

I can not reproduce the limit issue here. Just downloaded/installed your QGIS 2.4, launched it by double-click and ran the following in PyQGIS console:

for i in range(0,200):
  vlayer = QgsVectorLayer("/path/to/shapefile.shp", "layer {0}".format(i), "ogr")
  QgsMapLayerRegistry.instance().addMapLayer(vlayer)

For which, Activity Monitor shows 600 files loaded. I also saved and closed QGIS. Upon relaunching QGIS, the project was successfully auto-loaded (I have the option set).

I have also tested all my various dev builds here, via double-click. The qgis.log I have set shows the following:

Mac RLIMIT_NOFILE Soft/Hard NEW: 10240 / 10240
Mac RLIMIT_NOFILE Soft/Hard ORIG: 256 / 9223372036854775807

Will have to wait until next week to get access to the 10.7 nightly build VM to test it.

Is there maybe something funky with your Mac's setup? Have you tested on a different Mac as well?

qgib · 2014-07-04T13:06:10Z

Author Name: Larry Shaffer (Larry Shaffer)

William, I tested with 150 separate shapefiles, via your double-clicked 2.4 release install, to make sure repetitively loading the same shapefile wasn't masking the issue. Same results, everything loads fine, with Activity Monitor reporting 450 open shapefile components.

qgib · 2014-07-04T15:49:48Z

Author Name: William Kyngesburye (@kyngchaos)

Grr, something off with my Mac. Compiled on this Mac, OS X 10.8, with 10.7 SDK. Won't load > 84 shapefiles. Same problem in another login. Another Mac, OS X 10.7, it works. Yet another Mac 10.8, works.

qgib · 2015-12-03T05:25:45Z

Author Name: Médéric RIBREUX (Médéric RIBREUX)

Hello, bug triage...

I am able to open more than 250 Shapefile layers on QGIS 2.13 under Debian. Do you still have the same problem on OS X ?

If it is not the case, we could close this bug.

status_id was changed from Open to Feedback

qgib · 2015-12-03T06:18:23Z

Author Name: William Kyngesburye (@kyngchaos)

I upgraded the system on the problem Mac recently. I can now open more than 84 shapefiles. I never figured what in the old system (and just that Mac) was overriding the limits.

qgib · 2015-12-03T10:05:41Z

Author Name: Médéric RIBREUX (Médéric RIBREUX)

Okay, so I am closing this bug...

status_id was changed from Feedback to Closed

qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! Documentation Changes to Documentation, README, INSTALL, etc labels May 24, 2019

qgib added this to the Future Release - Lower Priority milestone May 24, 2019

qgib closed this as completed May 24, 2019

qgib mentioned this issue May 25, 2019

Too many layers -> unknown exception #15657

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Large Projects: add limitations to the manual #14000

Large Projects: add limitations to the manual #14000

qgib commented Jun 23, 2011

qgib commented Jun 27, 2011

qgib commented Jun 30, 2011

qgib commented Aug 29, 2011

qgib commented Sep 24, 2011

qgib commented Dec 16, 2011

qgib commented Apr 16, 2012

qgib commented Jun 21, 2012

qgib commented Jun 21, 2012

qgib commented Jun 22, 2012

qgib commented Jun 23, 2012

qgib commented Sep 4, 2012

qgib commented Jul 14, 2013

qgib commented Jul 14, 2013 •

edited

Loading

qgib commented Jul 20, 2013

qgib commented Jul 21, 2013

qgib commented Nov 3, 2013

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Dec 3, 2015

qgib commented Dec 3, 2015

qgib commented Dec 3, 2015

Large Projects: add limitations to the manual #14000

Large Projects: add limitations to the manual #14000

Comments

qgib commented Jun 23, 2011

qgib commented Jun 27, 2011

qgib commented Jun 30, 2011

qgib commented Aug 29, 2011

qgib commented Sep 24, 2011

qgib commented Dec 16, 2011

qgib commented Apr 16, 2012

qgib commented Jun 21, 2012

qgib commented Jun 21, 2012

qgib commented Jun 22, 2012

qgib commented Jun 23, 2012

qgib commented Sep 4, 2012

qgib commented Jul 14, 2013

qgib commented Jul 14, 2013 • edited Loading

qgib commented Jul 20, 2013

qgib commented Jul 21, 2013

qgib commented Nov 3, 2013

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jun 28, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Jul 4, 2014

qgib commented Dec 3, 2015

qgib commented Dec 3, 2015

qgib commented Dec 3, 2015

qgib commented Jul 14, 2013 •

edited

Loading