What is the maximum WCS export size #4421

Closed
msmitherdc opened this Issue Aug 13, 2012 · 19 comments

Comments

Projects
None yet
4 participants
@msmitherdc
Contributor

msmitherdc commented Aug 13, 2012

for a 64 bit instance

I'm getting messages like

agg2CreateImage(): Memory allocation error. mapagg.cpp: 796: Out of memory allocating 4164540520 bytes.

when making large (5 sq mile) wcs requests (1.0.0) against a tile index with resolution of (0.1 m).

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

Same problem here, but with regular WMS request.

MapServer:
MapServer version 6.2.0 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=GD SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_CLIENT SUPPORTS=THREADS SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE

Debian GNU/Linux 6.0.7 (squeeze)
64 bit
16 GB RAM

<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<ServiceExceptionReport version="1.3.0" xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
msDrawMap(): Image handling error. Unable to initialize image.
agg2CreateImage(): Memory allocation error. mapagg.cpp: 796: Out of memory allocating 3600000000 bytes.

</ServiceException>
</ServiceExceptionReport>

7o9 commented Oct 18, 2013

Same problem here, but with regular WMS request.

MapServer:
MapServer version 6.2.0 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=GD SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_CLIENT SUPPORTS=THREADS SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE

Debian GNU/Linux 6.0.7 (squeeze)
64 bit
16 GB RAM

<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<ServiceExceptionReport version="1.3.0" xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
msDrawMap(): Image handling error. Unable to initialize image.
agg2CreateImage(): Memory allocation error. mapagg.cpp: 796: Out of memory allocating 3600000000 bytes.

</ServiceException>
</ServiceExceptionReport>
@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

@7o9 what's the request URL ?

Member

tbonfort commented Oct 18, 2013

@7o9 what's the request URL ?

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

Works fine up about 23000 by 23000 pixels but we need up to ~70000 by 100000. Processing times is exceptionally fast, needs about 10 seconds for a map that needs 20 minutes rendering using ArcSomething... It is used for high resolution printing.

7o9 commented Oct 18, 2013

Works fine up about 23000 by 23000 pixels but we need up to ~70000 by 100000. Processing times is exceptionally fast, needs about 10 seconds for a map that needs 20 minutes rendering using ArcSomething... It is used for high resolution printing.

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

Is there anything else running on the server? i.e. do you have the 3.5 Gbytes of ram available for that allocation ?

Member

tbonfort commented Oct 18, 2013

Is there anything else running on the server? i.e. do you have the 3.5 Gbytes of ram available for that allocation ?

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

Hehe, yes, lots a virgin RAM waiting to be ... Ah, well. Yes.
bildschirmfoto vom 2013-10-18 13 06 20

7o9 commented Oct 18, 2013

Hehe, yes, lots a virgin RAM waiting to be ... Ah, well. Yes.
bildschirmfoto vom 2013-10-18 13 06 20

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

also note that your 70000*100000 requests are each going to require 28Gb of memory, which is clearly going to fail on a 16Gb server

Member

tbonfort commented Oct 18, 2013

also note that your 70000*100000 requests are each going to require 28Gb of memory, which is clearly going to fail on a 16Gb server

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

@7o9 maybe check your ulimits ?

Member

tbonfort commented Oct 18, 2013

@7o9 maybe check your ulimits ?

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

What is the formula for calculating the required RAM?

7o9 commented Oct 18, 2013

What is the formula for calculating the required RAM?

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

width_height_4 , result in bytes

Member

tbonfort commented Oct 18, 2013

width_height_4 , result in bytes

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

Thanks. We can adjust resolution to fit the server / stock up on RAM. But this does not solve the original problem. The request only requires 3.6 GB as per your formula and there are definitely ~15GB available?!

7o9 commented Oct 18, 2013

Thanks. We can adjust resolution to fit the server / stock up on RAM. But this does not solve the original problem. The request only requires 3.6 GB as per your formula and there are definitely ~15GB available?!

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

After some playing around I think we have a hard limit of 2GB somewhere in MapServer. I can request up to 23k square images, anything more will break things. Makes sense?

7o9 commented Oct 18, 2013

After some playing around I think we have a hard limit of 2GB somewhere in MapServer. I can request up to 23k square images, anything more will break things. Makes sense?

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

yes, makes sense... that's what my "check your ulimits" idea was for

Member

tbonfort commented Oct 18, 2013

yes, makes sense... that's what my "check your ulimits" idea was for

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

Ah, I misunderstood you then. I thought you suggested limits of image size... What is "ulimits" & where can I check / set them (outing myself as a noob).

7o9 commented Oct 18, 2013

Ah, I misunderstood you then. I thought you suggested limits of image size... What is "ulimits" & where can I check / set them (outing myself as a noob).

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 18, 2013

Member

I've never tinkered with them personally. check out https://wiki.debian.org/Limits , and more precisely
For example, apache2 limits are configured in /etc/apache2/envvars

Member

tbonfort commented Oct 18, 2013

I've never tinkered with them personally. check out https://wiki.debian.org/Limits , and more precisely
For example, apache2 limits are configured in /etc/apache2/envvars

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 18, 2013

...later, rechecked all:

MapServer is 64bit binary running on a 64 bit system with sufficient RAM.

The Apache processes are running as user www-data and:
www-data@ns3100632:~$ ulimit -v
returns "unlimited"

But we do get a segmentation fault:
mapserv-6.2[5824]: segfault at 7f5943f76008 ip 00007f5a7045fb39 sp 00007fffa54b4880 error 4 in libc-2.11.3.so[7f5a703e9000+159000]

So I still suspect that it is some hard-coded limit in MapServer or one of its libraries.

7o9 commented Oct 18, 2013

...later, rechecked all:

MapServer is 64bit binary running on a 64 bit system with sufficient RAM.

The Apache processes are running as user www-data and:
www-data@ns3100632:~$ ulimit -v
returns "unlimited"

But we do get a segmentation fault:
mapserv-6.2[5824]: segfault at 7f5943f76008 ip 00007f5a7045fb39 sp 00007fffa54b4880 error 4 in libc-2.11.3.so[7f5a703e9000+159000]

So I still suspect that it is some hard-coded limit in MapServer or one of its libraries.

@7o9

This comment has been minimized.

Show comment
Hide comment
@7o9

7o9 Oct 19, 2013

This issue is very easy to reproduce, simply call a WMS request with a width and height exceeding 25000 pixel. Always works, like in: always breaks.

7o9 commented Oct 19, 2013

This issue is very easy to reproduce, simply call a WMS request with a width and height exceeding 25000 pixel. Always works, like in: always breaks.

rouault added a commit to rouault/mapserver that referenced this issue Oct 19, 2013

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Oct 20, 2013

Member

@7o9, please check #4796 for a possible fix. cc @rouault

Member

tbonfort commented Oct 20, 2013

@7o9, please check #4796 for a possible fix. cc @rouault

@jratike80

This comment has been minimized.

Show comment
Hide comment
@jratike80

jratike80 Oct 28, 2015

Questions belong IMHO better to mapserver-users mailing list. Three years later I asked the same question because I do not follow Github issues so closely.
http://osgeo-org.1560.x6.nabble.com/What-resourse-blocks-any-bigger-GeoTIFF-output-from-WCS-td5218584.html

I suppose that this ticket does not deal with any exact bug so I am closing it.

Questions belong IMHO better to mapserver-users mailing list. Three years later I asked the same question because I do not follow Github issues so closely.
http://osgeo-org.1560.x6.nabble.com/What-resourse-blocks-any-bigger-GeoTIFF-output-from-WCS-td5218584.html

I suppose that this ticket does not deal with any exact bug so I am closing it.

@jratike80 jratike80 closed this Oct 28, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment