Skip to content

What is the maximum WCS export size #4421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
msmitherdc opened this issue Aug 13, 2012 · 19 comments
Closed

What is the maximum WCS export size #4421

msmitherdc opened this issue Aug 13, 2012 · 19 comments

Comments

@msmitherdc
Copy link
Contributor

for a 64 bit instance

I'm getting messages like

agg2CreateImage(): Memory allocation error. mapagg.cpp: 796: Out of memory allocating 4164540520 bytes.

when making large (5 sq mile) wcs requests (1.0.0) against a tile index with resolution of (0.1 m).

@7o9
Copy link

7o9 commented Oct 18, 2013

Same problem here, but with regular WMS request.

MapServer:
MapServer version 6.2.0 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG SUPPORTS=PROJ SUPPORTS=GD SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=ICONV SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_CLIENT SUPPORTS=THREADS SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE

Debian GNU/Linux 6.0.7 (squeeze)
64 bit
16 GB RAM

<?xml version='1.0' encoding="ISO-8859-1" standalone="no" ?>
<ServiceExceptionReport version="1.3.0" xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
msDrawMap(): Image handling error. Unable to initialize image.
agg2CreateImage(): Memory allocation error. mapagg.cpp: 796: Out of memory allocating 3600000000 bytes.

</ServiceException>
</ServiceExceptionReport>

@tbonfort
Copy link
Member

@7o9 what's the request URL ?

@7o9
Copy link

7o9 commented Oct 18, 2013

Works fine up about 23000 by 23000 pixels but we need up to ~70000 by 100000. Processing times is exceptionally fast, needs about 10 seconds for a map that needs 20 minutes rendering using ArcSomething... It is used for high resolution printing.

@tbonfort
Copy link
Member

Is there anything else running on the server? i.e. do you have the 3.5 Gbytes of ram available for that allocation ?

@7o9
Copy link

7o9 commented Oct 18, 2013

Hehe, yes, lots a virgin RAM waiting to be ... Ah, well. Yes.
bildschirmfoto vom 2013-10-18 13 06 20

@tbonfort
Copy link
Member

also note that your 70000*100000 requests are each going to require 28Gb of memory, which is clearly going to fail on a 16Gb server

@tbonfort
Copy link
Member

@7o9 maybe check your ulimits ?

@7o9
Copy link

7o9 commented Oct 18, 2013

What is the formula for calculating the required RAM?

@tbonfort
Copy link
Member

width_height_4 , result in bytes

@7o9
Copy link

7o9 commented Oct 18, 2013

Thanks. We can adjust resolution to fit the server / stock up on RAM. But this does not solve the original problem. The request only requires 3.6 GB as per your formula and there are definitely ~15GB available?!

@7o9
Copy link

7o9 commented Oct 18, 2013

After some playing around I think we have a hard limit of 2GB somewhere in MapServer. I can request up to 23k square images, anything more will break things. Makes sense?

@tbonfort
Copy link
Member

yes, makes sense... that's what my "check your ulimits" idea was for

@7o9
Copy link

7o9 commented Oct 18, 2013

Ah, I misunderstood you then. I thought you suggested limits of image size... What is "ulimits" & where can I check / set them (outing myself as a noob).

@tbonfort
Copy link
Member

I've never tinkered with them personally. check out https://wiki.debian.org/Limits , and more precisely
For example, apache2 limits are configured in /etc/apache2/envvars

@7o9
Copy link

7o9 commented Oct 18, 2013

...later, rechecked all:

MapServer is 64bit binary running on a 64 bit system with sufficient RAM.

The Apache processes are running as user www-data and:
www-data@ns3100632:~$ ulimit -v
returns "unlimited"

But we do get a segmentation fault:
mapserv-6.2[5824]: segfault at 7f5943f76008 ip 00007f5a7045fb39 sp 00007fffa54b4880 error 4 in libc-2.11.3.so[7f5a703e9000+159000]

So I still suspect that it is some hard-coded limit in MapServer or one of its libraries.

@7o9
Copy link

7o9 commented Oct 19, 2013

This issue is very easy to reproduce, simply call a WMS request with a width and height exceeding 25000 pixel. Always works, like in: always breaks.

rouault added a commit to rouault/mapserver that referenced this issue Oct 19, 2013
@tbonfort
Copy link
Member

@7o9, please check #4796 for a possible fix. cc @rouault

@jratike80
Copy link

Questions belong IMHO better to mapserver-users mailing list. Three years later I asked the same question because I do not follow Github issues so closely.
http://osgeo-org.1560.x6.nabble.com/What-resourse-blocks-any-bigger-GeoTIFF-output-from-WCS-td5218584.html

I suppose that this ticket does not deal with any exact bug so I am closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants