New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preventing DOS situation with WMS-T service specifying effective huge TIME range #4524

Closed
akrherz opened this Issue Nov 13, 2012 · 8 comments

Comments

Projects
None yet
3 participants
@akrherz
Contributor

akrherz commented Nov 13, 2012

I'm running the WMS-T service shown as the example in the MS documentation here :)

http://mapserver.org/ogc/wms_time.html#wms-time

The problem is that when a user specifies a CGI variable of 'TIME=2011', the database query ends up returning every raster for the year 2011 (a raster every 5 minutes). This causes mapserver to do a lot of work and eventually DOSs the server...

The documentation notes that setting the metadata wms_timeformat should help limit the potential timestamp queries coming to the server, but this feature may not be implemented for PostGIS backed WMS-T queries?

Note that this functionality is only available on layers based on Shapefiles
and OGR.

tbonfort kindly helped me with this on IRC and asked if I might have a suggestion on how to deal with this. I am not sure if the issue here is if the postgis backend is not supporting the wms_timeformat setting and implementing that would resolve this or if some syntax to the SQL query is needed to limit the number of rows that could be potentially returned. In my mind, the wms_timeformat would be best and I would want to make sure a request has at least the minutes specified in the TIME CGI variable.

thanks

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Nov 13, 2012

Contributor

I got 6.2.0-rc running and debugged the resulting SQL from just a &TIME=2010

select "datetime",encode(ST_AsBinary(ST_Force_2D("the_geom"),'NDR'),'hex') as geom,"oid" 
from nexrad_n0r_tindex where the_geom && 
ST_GeomFromText(<snip>,find_srid('','nexrad_n0r_tindex','the_geom')) and 
(datetime >= date_trunc('year',date '2010-01-01') and 
datetime < date_trunc('year',date '2010-01-01') + interval '1 year') 
and (datetime >= date_trunc('year',date '2010-01-01') and 
datetime < date_trunc('year',date '2010-01-01') + interval '1 year')

interesting. Still digging :)

Contributor

akrherz commented Nov 13, 2012

I got 6.2.0-rc running and debugged the resulting SQL from just a &TIME=2010

select "datetime",encode(ST_AsBinary(ST_Force_2D("the_geom"),'NDR'),'hex') as geom,"oid" 
from nexrad_n0r_tindex where the_geom && 
ST_GeomFromText(<snip>,find_srid('','nexrad_n0r_tindex','the_geom')) and 
(datetime >= date_trunc('year',date '2010-01-01') and 
datetime < date_trunc('year',date '2010-01-01') + interval '1 year') 
and (datetime >= date_trunc('year',date '2010-01-01') and 
datetime < date_trunc('year',date '2010-01-01') + interval '1 year')

interesting. Still digging :)

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Nov 13, 2012

Contributor

I think the problem is that msSetLimitedPattersToUse() is never getting called, because the ms_limited_pattern is not getting set early enough. The only place it gets set is within msSetLimitedPattersToUse() . I think this circular logic is the problem , but my head is spinning at the moment.

Contributor

akrherz commented Nov 13, 2012

I think the problem is that msSetLimitedPattersToUse() is never getting called, because the ms_limited_pattern is not getting set early enough. The only place it gets set is within msSetLimitedPattersToUse() . I think this circular logic is the problem , but my head is spinning at the moment.

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Feb 7, 2013

Contributor

Here's a test case.

ms4524.sql

create table ms4524(
   datetime timestamptz,
   filepath varchar
);
ALTER TABLE ms4524 SET WITH oids;
SELECT AddGeometryColumn('ms4524', 'geom', 4326, 'MULTIPOLYGON', 2);
GRANT SELECT on ms4524 to apache;

INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:00+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 
INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:05+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 
INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:10+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 

ms4524.map

MAP
#
# Start of map file
#
NAME "nexrad_base_reflect"
STATUS ON
SIZE 800 600
EXTENT -171 15 -66 70
UNITS DD
IMAGECOLOR -1 -1 -1

PROJECTION
   "init=epsg:4326"
END

WEB
  MINSCALE 90000
  MAXSCALE 4650000
  IMAGEPATH "/tmp/"
  IMAGEURL "/tmp/"
  LOG "/tmp/ms4524.log"
  METADATA
   "wms_title" "IEM WMS Service"
   "wms_onlineresource" "http://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0q-t.cgi?"
   "wms_srs" "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
   "wms_abstract" "IEM generated CONUS composite of NWS WSR-88D level III base reflectivity."
   "wms_keywords" "NEXRAD,N0Q"
   "wms_accessconstraints" "None"
   "wms_contactperson" "Daryl Herzmann"
   "wms_contactorganization" "Iowa State University"
   "wms_enable_request" "*"
  END
END

#  Tile Index
LAYER
  STATUS ON
  NAME "time_idx"
  TYPE POLYGON
  DATA "geom from ms4524"
  FILTER "datetime = '2006-06-23 03:10+00'"
  METADATA
    "wms_title" "NEXRAD BASE REFLECT"
    "wms_srs"   "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
    "wms_extent" "-171 15 -66 70"
    "wms_timeextent" "2011-02-16/2013-12-31/PT5M"
    "wms_timeitem" "datetime"
    "wms_timedefault" "2011-02-21T19:30:00Z"
    "wms_timeformat" "YYYY-MM-DDTHH:MM:SSZ"
  END
  CONNECTION "dbname=postgis"
  CONNECTIONTYPE postgis
END

# raster layer
LAYER
  NAME "nexrad-n0q-wmst"
  TYPE RASTER
  STATUS ON
  DEBUG ON
  DUMP TRUE
  PROJECTION
    "init=epsg:4326"
  END
  METADATA
    "wms_title" "NEXRAD BASE REFLECT"
    "wms_srs"   "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
    "wms_extent" "-171 15 -66 70"
    "wms_timeextent" "2011-02-16/2013-12-31/PT5M"
    "wms_timeitem" "datetime"
    "wms_timedefault" "2011-02-21T19:30:00Z"
    "wms_timeformat" "YYYY-MM-DDTHH:MM:SSZ"
  END
  OFFSITE 0 0 0
  TILEITEM "filepath"
  TILEINDEX "time_idx"
END

END

Example request:

http://localhost/cgi-bin/mapserv/mapserv?map=/tmp/ms4524.map&LAYERS=nexrad-n0q-wmst&TRANSPARENT=TRUE&FORMAT=image%2Fpng&TIME=2013&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG%3A4326&BBOX=-67.5,0,-45,22.5&WIDTH=256&HEIGHT=256

resulting sql

select "datetime","filepath",encode(ST_AsBinary(ST_Force_2D("geom"),'NDR'),'hex') as geom,"oid" from ms4524 where geom && ST_GeomFromText('POLYGON((-67.4560546875 0.0439453125,-67.4560546875 22.4560546875,-45.0439453125 22.4560546875,-45.0439453125 0.0439453125,-67.4560546875 0.0439453125))',find_srid('','ms4524','geom')) and (datetime >= date_trunc('year',date '2013-01-01') and datetime < date_trunc('year',date '2013-01-01') + interval '1 year')

thanks

Contributor

akrherz commented Feb 7, 2013

Here's a test case.

ms4524.sql

create table ms4524(
   datetime timestamptz,
   filepath varchar
);
ALTER TABLE ms4524 SET WITH oids;
SELECT AddGeometryColumn('ms4524', 'geom', 4326, 'MULTIPOLYGON', 2);
GRANT SELECT on ms4524 to apache;

INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:00+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 
INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:05+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 
INSERT into ms4524(datetime, filepath, geom) VALUES 
  ('2013-01-01 00:10+00', '/tmp/bogus.png', 
  ST_GeomFromText('MULTIPOLYGON(((-126 50,-66 50,-66 24,-126 24,-126 50)))',
    4326)); 

ms4524.map

MAP
#
# Start of map file
#
NAME "nexrad_base_reflect"
STATUS ON
SIZE 800 600
EXTENT -171 15 -66 70
UNITS DD
IMAGECOLOR -1 -1 -1

PROJECTION
   "init=epsg:4326"
END

WEB
  MINSCALE 90000
  MAXSCALE 4650000
  IMAGEPATH "/tmp/"
  IMAGEURL "/tmp/"
  LOG "/tmp/ms4524.log"
  METADATA
   "wms_title" "IEM WMS Service"
   "wms_onlineresource" "http://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0q-t.cgi?"
   "wms_srs" "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
   "wms_abstract" "IEM generated CONUS composite of NWS WSR-88D level III base reflectivity."
   "wms_keywords" "NEXRAD,N0Q"
   "wms_accessconstraints" "None"
   "wms_contactperson" "Daryl Herzmann"
   "wms_contactorganization" "Iowa State University"
   "wms_enable_request" "*"
  END
END

#  Tile Index
LAYER
  STATUS ON
  NAME "time_idx"
  TYPE POLYGON
  DATA "geom from ms4524"
  FILTER "datetime = '2006-06-23 03:10+00'"
  METADATA
    "wms_title" "NEXRAD BASE REFLECT"
    "wms_srs"   "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
    "wms_extent" "-171 15 -66 70"
    "wms_timeextent" "2011-02-16/2013-12-31/PT5M"
    "wms_timeitem" "datetime"
    "wms_timedefault" "2011-02-21T19:30:00Z"
    "wms_timeformat" "YYYY-MM-DDTHH:MM:SSZ"
  END
  CONNECTION "dbname=postgis"
  CONNECTIONTYPE postgis
END

# raster layer
LAYER
  NAME "nexrad-n0q-wmst"
  TYPE RASTER
  STATUS ON
  DEBUG ON
  DUMP TRUE
  PROJECTION
    "init=epsg:4326"
  END
  METADATA
    "wms_title" "NEXRAD BASE REFLECT"
    "wms_srs"   "EPSG:4326 EPSG:900913 EPSG:102100 EPSG:3857"
    "wms_extent" "-171 15 -66 70"
    "wms_timeextent" "2011-02-16/2013-12-31/PT5M"
    "wms_timeitem" "datetime"
    "wms_timedefault" "2011-02-21T19:30:00Z"
    "wms_timeformat" "YYYY-MM-DDTHH:MM:SSZ"
  END
  OFFSITE 0 0 0
  TILEITEM "filepath"
  TILEINDEX "time_idx"
END

END

Example request:

http://localhost/cgi-bin/mapserv/mapserv?map=/tmp/ms4524.map&LAYERS=nexrad-n0q-wmst&TRANSPARENT=TRUE&FORMAT=image%2Fpng&TIME=2013&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG%3A4326&BBOX=-67.5,0,-45,22.5&WIDTH=256&HEIGHT=256

resulting sql

select "datetime","filepath",encode(ST_AsBinary(ST_Force_2D("geom"),'NDR'),'hex') as geom,"oid" from ms4524 where geom && ST_GeomFromText('POLYGON((-67.4560546875 0.0439453125,-67.4560546875 22.4560546875,-45.0439453125 22.4560546875,-45.0439453125 0.0439453125,-67.4560546875 0.0439453125))',find_srid('','ms4524','geom')) and (datetime >= date_trunc('year',date '2013-01-01') and datetime < date_trunc('year',date '2013-01-01') + interval '1 year')

thanks

mkofahl pushed a commit to faegi/mapserver that referenced this issue Apr 9, 2013

@ghost ghost assigned aboudreault Apr 13, 2013

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Apr 14, 2014

Contributor

With mapserver 6.4.1, it appears this request is able to bring about the DOS type situation as well

TIME=2006-12-31T17:36:00Z/2008-12-31T03:12:00Z

Contributor

akrherz commented Apr 14, 2014

With mapserver 6.4.1, it appears this request is able to bring about the DOS type situation as well

TIME=2006-12-31T17:36:00Z/2008-12-31T03:12:00Z

@aboudreault

This comment has been minimized.

Show comment
Hide comment
@aboudreault

aboudreault Apr 15, 2014

Member

@dmorissette you might want to take a look at this.

Member

aboudreault commented Apr 15, 2014

@dmorissette you might want to take a look at this.

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Jun 15, 2015

Contributor

Any updates on this issue? I am testing out current master and see the DOS condition is still there :(

Contributor

akrherz commented Jun 15, 2015

Any updates on this issue? I am testing out current master and see the DOS condition is still there :(

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Jun 17, 2015

Member

I still believe there is nothing much we can do on the mapserver side, as from what I understand its behavior in this case is correct. The logic of ordering and limiting the number of features from the tileindex is very specific to each scenario, and can't really be hardcoded into mapserver. I suggest you either use MAXFEATURES on your tileindex layer, or use a LIMIT+ORDER BY clause in your sql query.

Member

tbonfort commented Jun 17, 2015

I still believe there is nothing much we can do on the mapserver side, as from what I understand its behavior in this case is correct. The logic of ordering and limiting the number of features from the tileindex is very specific to each scenario, and can't really be hardcoded into mapserver. I suggest you either use MAXFEATURES on your tileindex layer, or use a LIMIT+ORDER BY clause in your sql query.

@akrherz

This comment has been minimized.

Show comment
Hide comment
@akrherz

akrherz Jun 17, 2015

Contributor

@tbonfort Thank you for your response. For your suggestions, I could not make MAXFEATURES work and I found no method to inject a LIMIT clause into the DATA setting without breaking the layer entirely. Could you kindly suggest a DATA setting that you had in mind?

Perhaps a PROCESSING or METADATA option could be added to disable this functionality entirely? With 6.4.1, I simply made this change

--- mappostgis.c-orig   2015-06-17 07:59:44.793284497 -0500
+++ mappostgis.c    2015-06-15 14:28:40.717011966 -0500
@@ -3258,7 +3258,8 @@
         strlcat(buffer, bufferTmp, buffer_size);
       } else if(numranges == 2) {
         /* we have a range */
-        createPostgresTimeCompareRange(timefield, aranges[0], aranges[1], bufferTmp, buffer_size);
+        createPostgresTimeCompareSimple(timefield, atimes[i], bufferTmp, buffer_size);
+        //createPostgresTimeCompareRange(timefield, aranges[0], aranges[1], bufferTmp, buffer_size);
         strlcat(buffer, bufferTmp, buffer_size);
       } else {
         return MS_FALSE;

I see the code has changed for 7.0.0, so will try to manually hack it out of the code again.

EDIT: I got MAXFEATURES to work, thanks.

Contributor

akrherz commented Jun 17, 2015

@tbonfort Thank you for your response. For your suggestions, I could not make MAXFEATURES work and I found no method to inject a LIMIT clause into the DATA setting without breaking the layer entirely. Could you kindly suggest a DATA setting that you had in mind?

Perhaps a PROCESSING or METADATA option could be added to disable this functionality entirely? With 6.4.1, I simply made this change

--- mappostgis.c-orig   2015-06-17 07:59:44.793284497 -0500
+++ mappostgis.c    2015-06-15 14:28:40.717011966 -0500
@@ -3258,7 +3258,8 @@
         strlcat(buffer, bufferTmp, buffer_size);
       } else if(numranges == 2) {
         /* we have a range */
-        createPostgresTimeCompareRange(timefield, aranges[0], aranges[1], bufferTmp, buffer_size);
+        createPostgresTimeCompareSimple(timefield, atimes[i], bufferTmp, buffer_size);
+        //createPostgresTimeCompareRange(timefield, aranges[0], aranges[1], bufferTmp, buffer_size);
         strlcat(buffer, bufferTmp, buffer_size);
       } else {
         return MS_FALSE;

I see the code has changed for 7.0.0, so will try to manually hack it out of the code again.

EDIT: I got MAXFEATURES to work, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment