OGR Datasource is loaded although scale-denominator prohibits to render

[NasNasingud]

I render maps from XML style files having two layers. Both layers are based on an OGR/WFS datasource. One of the layers contains much data (~ 30000 polygons, let's call it heavylayer), the other layer is lightweight (~ 200 points, let's call it lightlayer). The lightlayer should be rendered at every scale, whereas the heavylayer should be rendered only at specific scale levels (e.g. maximum-scale-denominator < 50000).

I have attached an exemplary XML file as TXT.

The issue is that mapnik seems to fetch the data from both layers at every scale, regardless if the maximum-scale-denominator prohibits the heavylayer from being rendered. The map itself looks fine, heavylayer is not rendered at every scale, but it takes much time to create the map, even if only the lightlayer is rendered. When I, however, disable the heavylayer manually (status=off), then the data is not fetched and the map is rendered quickly.

As a test, I have rendered 10 maps and averaged the result:

heavylayer status=on, should be rendered according to scale-denominator: 15.43 s
heavylayer status=on, should NOT be rendered according to scale-denominator: 14.02 s (!!)
heavylayer status=off: 0.18 s

Interestingly, when using a Shapefile as datasource, it works fine (besides being also much faster):

heavylayer status=on, should be rendered according to scale-denominator: 0.28 s
heavylayer status=on, should NOT be rendered according to scale-denominator: 0.04 s
heavylayer status=off: 0.03 s

I am running mapnik 3.0.18 on ubuntu 17.10 with GDAL 2.2.1.

Do you have an idea about this issue? Thank you!
test.txt

[talaj]

The scale-denominator limit should work regardless of datasource type. Do you include also XML style load time to the timings?

[NasNasingud]

Yes, the XML style load is included in the timing. However, the length of the XML style file is only 38 lines (1 KB, see attached test.txt), so it should not be a big issue for the timing.

In the meantime I have made another interesting discovery. I checked also the timings for using a Shapefile, but this time reading it via OGR. The results:

heavylayer status=on, should be rendered according to scale-denominator: 0.68 s
heavylayer status=on, should NOT be rendered according to scale-denominator: 0.05 s
heavylayer status=off: 0.04 s

Hence, Shapefiles via OGR seem to work fine. The issue can obviously be further limited to other OGR sources...

[talaj]

I would not add XML style load time and rendering time together. They are distinct phases and distinct API calls. Style is commonly loaded once, then there can be arbitrary number of render calls.
This is the reason why the datasource is loaded regardless of scale-denominator, because there is no prior information that rendering of the layer will not take place.

[NasNasingud]

Thank you for the explanation! That means: when I switch the layer off manually, the datasource is not loaded at all. If the layer is switched on, then all datasources are loaded, regardless if being rendered later or not, and the decision about being rendered based on the scale-denominator is made later in the rendering step. Is this correct?
Hence, to improve the speed of rendering from slow WFS datasources, I would have to dynamically determine beforehand which layers have to be rendered at the given scale (e.g. with some Python code) and then switch the layers dynamically on and off?

[talaj]

If the layer is switched on, then all datasources are loaded, regardless if being rendered later or not, and the decision about being rendered based on the scale-denominator is made later in the rendering step. Is this correct?

Yes, this is correct. There is no render-time information about scale-denominator during style load.

Hence, to improve the speed of rendering from slow WFS datasources, I would have to dynamically determine beforehand which layers have to be rendered at the given scale (e.g. with some Python code) and then switch the layers dynamically on and off?

No, I would not do that. I think that speed of rendering is good for both type of datasources. At least this is an impression I have from timings you have provided. Speed of style loading is the thing what is slow here and it does not matter in most cases. Do you have to load style for every rendering request? I can only imagine this usage for style development.

The reason for difference in datasource initialization time between Shapefile and OGR WFS can lay outside of Mapnik. I'm not an expert on OGR datasource, but after a quick look into the code, the datasource is querying OGR driver for extent of all features. This operation can take some time if the value is not cached somehow. You can set extent parameter to the OGR datasource to avoid potentially slow extent computation.

[NasNasingud]

Speed of style loading is the thing what is slow here

Yes, definitely. The rendering itself is fast, the OGR WFS is the bottleneck. However, I didn't know that Mapnik first fetches all datasources and then, in the second step, discards those who are not needed. This was the main point of this issue - thanks for clarifying!

Do you have to load style for every rendering request?

Yes. Currently, I use Mapnik as rendering machine for a WMS service. For each WMS request, the style is loaded - and hence also all datasources. This is also done because the original data may have changed.

You can set extent parameter to the OGR datasource

Thank you for this tip. Although I suspect that OGR fetches the extent after the first request from the cache, because the result of the GetCapabilities request is then appended to the XML connection file (http://www.gdal.org/drv_wfs.html).