Improved numerical validation. by sdlime · Pull Request #6458 · MapServer/MapServer

sdlime · 2022-01-24T00:17:52Z

Implements a common approach to validating numerical values read from the mapfile at parse time. Meant to complement runtime validation done throughout the code. Review and additions most welcome.

This reverts commit 7d261af.

…e (matches Cairo approach).

geographika · 2022-01-24T09:51:19Z

mapfile.c

        if((symbol = getSymbol(2, MS_NUMBER,MS_BINDING)) == -1) return(-1);
        if(symbol == MS_NUMBER) {
-          label->shadowsizex = (int) msyynumber;
+          label->shadowsizex = (int) msyynumber; // TODO > 0


Are these TODOs due to complications around adding in the checks in these cases?
In similar cases below they are commented "ok?"

No, this is an oversight on my part - forgot to resolve the "TODO". Will update...

geographika · 2022-01-24T09:55:36Z

mapfile.c

      case(SIZE):
-        if(getInteger(&(ref->width)) == -1) return(-1);
-        if(getInteger(&(ref->height)) == -1) return(-1);
+        if(getInteger(&(ref->width), MS_NUM_CHECK_RANGE, 5, ref->map->maxsize) == -1) return(-1); // is 5 reasonable?


Should the minimum values of 5 be added to https://mapserver.org/mapfile/reference.html#mapfile-reference-size ?

Yes. Once there is agreement that that's the appropriate value. It seemed reasonable to me...

geographika · 2022-01-24T09:58:14Z

mapfile.c

      case(ANGLE): {
        double rotation_angle;
-        if(getDouble(&(rotation_angle)) == -1) return MS_FAILURE;
+        if(getDouble(&(rotation_angle), MS_NUM_CHECK_RANGE, -360, 360) == -1) return MS_FAILURE;


Can a https://mapserver.org/mapfile/map.html#mapfile-map-angle MAP->ANGLE have a negative value?

I believe so - I mean it makes sense to me. @tbonfort, what do you think?

geographika · 2022-01-24T10:02:03Z

mapfile.c

-int loadQueryMap(queryMapObj *querymap)
+int loadQueryMap(queryMapObj *querymap, mapObj *map)
 {
+  querymap->map = (mapObj *)map;


Could there be any issues with adding the map as a property to the querymap? Would it be safer to pass in the maxsize and minsize values directly to this function?

We use the same approach with other objects (e.g. layer, scalebar) where there is an opaque pointer back to the parent so this should be ok I'd think so I was sticking to that pattern. Maybe @rouault has an opinion on this practice.

geographika · 2022-01-24T10:03:28Z

mapfile.c

          msyylex();

-          if(getDouble(&(rotation_angle)) == -1) break;
+          if(getDouble(&(rotation_angle), MS_NUM_CHECK_RANGE, -360, 360) == -1) break;


As above, I was unaware you could use negative angles (to presumably set anti-clockwise rotation?)

Would be good to get confirmation from @tbonfort but this should be ok and should be easy to test.

geographika · 2022-01-24T10:06:55Z

mapserver.h

+#ifdef SWIG
+    %immutable;
+#endif /* SWIG */
+    struct mapObj *map; ///< Reference to parent :class:`mapObj`                                                                                                                 


May not be required if only used to get the min and max size when loading the map (see comment above).

geographika · 2022-01-24T10:07:38Z

mapserver.h

+  #define MS_SCALEBAR_INTERVALS_MIN 1
+  #define MS_SCALEBAR_INTERVALS_MAX 100
+
+  #define MS_SCALEBAR_WIDTH_MIN 5


Docs to udpate at https://mapserver.org/mapfile/scalebar.html#mapfile-scalebar-size

Yes, once values are confirmed.

geographika · 2022-01-24T10:08:17Z

mapserver.h

  /*                              legendObj                               */
  /************************************************************************/

+  #define MS_LEGEND_KEYSIZE_MIN 5


Docs to update at https://mapserver.org/mapfile/legend.html

Yes, once values are confirmed.

geographika · 2022-01-24T13:44:26Z

@sdlime - looks good to me, and any validation that can be done at the Mapfile parsing stage will save a lot of pain later on.

I ran on a couple of large Mapfiles I have here, and hit the error message loadStyle(): General error message. Invalid WIDTH, must be greater than 0. On further investigation this was correctly thrown as I had a STYLE with a WIDTH 0 which serves no purpose.

CLASS
..
            STYLE
                COLOR 0 0 0
                WIDTH 0
            END
            STYLE
                COLOR 255 192 203
                WIDTH 4
            END
        END
END

Do you expect parsing speeds to be impacted in any way? I can try and test that here to see.

Finally I can look at making a docs pull request with the min/max values for various keywords if that helps.

sdlime · 2022-01-25T17:39:42Z

@geographika, I'm not anticipating any significant impact on parsing speed. If you have the opportunity to validate then that would be great. In the big scheme of things additions like the config file will have a more significant affect - especially for straight CGI execution.

In terms of things like WIDTH 0, I wrestled with handling default values when I worked through this. I did do some testing when writing mapfiles via MapScript to make sure that default values weren't being written routinely. I mean, we wouldn't want MapServer to barf reading a mapfile that was written by MapServer. That doesn't seem to be the case here and we can recommend commenting things out rather than using a value like 0.

That said, we could conditionally allow default values in a few cases if something like WIDTH 0 is pretty commonplace - recommended in the docs or whatever.

--Steve

mapfile.c

…d clarifying comments in loadColor(). Resolved a couple of TODO comments.

sdlime · 2022-01-26T13:45:17Z

I think I got things resolved (except for docs)... Please let me know about merging. --Steve

sdlime · 2022-01-26T13:53:10Z

This will be a candidate for backport to 7.6 but I think we should wait until some 8.0 testing happens.

geographika · 2022-01-26T13:55:44Z

@sdlime - do you want me to update the docs? I'm also going to update mappyfile to include the min/max values so can do both at the same time.

sdlime · 2022-01-26T13:56:48Z

@sdlime - do you want me to update the docs? I'm also going to update mappyfile to include the min/max values so can do both at the same time.

That would be great!

jmckenna · 2022-01-26T14:07:57Z

re: backport: I'd also prefer getting feedback through main branch first (will do tests with MS4W today)

re: docs: I'd like to see us getting back to filing an associated ticket in the documentation repository, for each new feature: the benefit is that the docs team will test each new feature, while adding the docs. This also is a way to keep track of what is missing in the docs.

ejn · 2022-01-26T14:21:25Z

As a user I'd say that this change (much as I find it a good thing (TM)) shouldn't be backported - it's a change that could break existing installations. Yes, those are installations with "invalid" mapfiles, but if they weren't sufficiently invalid that they couldn't be used then it would be rude to introduce what is effectively a breaking change in a minor (or even bugfix) release.

jmckenna · 2022-01-26T14:33:23Z

I agree with you @ejn

jmckenna · 2022-01-26T18:13:15Z

@sdlime something bizarre is happening locally for me, sorry if this sounds like a crazy question: could the output of map2img's -map_debug switch be affected by these changes? -all_debug 3 works fine. -map_debug 3 generates the map image but does not output any debug messages (just a blank line). This works with a build from a month ago. Something is different, but I don't know what yet. (either in my build, or a change in the code)

sdlime · 2022-01-26T18:58:02Z

No, the various debug settings are unaffected. This is actually an oversight since if debug is given as a number then it should be checked as well but the code in mapfile.c uses getSymbol() so I missed it. I can patch that...

Anyway, map2img is setting map->debug directly, apart from what's in the mapfile so these changes have no bearing on that. I'm happy to try on my end if you have a test case.

jmckenna · 2022-01-26T19:01:28Z

Ok thanks, makes sense. I'll take a deeper look and see if it was something on my end (likely) first.

jmckenna · 2022-01-27T13:47:55Z

@sdlime solved, it was on my end. Will continue testing this...

jmckenna · 2022-01-27T17:22:37Z

@Steve can something be done to include a line number in the error message? For example, a mapfile with 30 entries of WIDTH 0 returns only the following with map2img:

loadStyle(): General error message. Invalid WIDTH, must be greater than 0
loadStyle(): General error message. Invalid WIDTH, must be greater than 0 <br>

Or is this a huge can of worms, for another day?

sdlime · 2022-01-27T18:24:46Z

@Steve can something be done to include a line number in the error message? For example, a mapfile with 30 entries of WIDTH 0 returns only the following with map2img:
loadStyle(): General error message. Invalid WIDTH, must be greater than 0
loadStyle(): General error message. Invalid WIDTH, must be greater than 0 <br>
Or is this a huge can of worms, for another day?

No, this can be addressed in the error handling. That said, we could also test for >= 0 if width=0 is a common setting (the default width is 1). @geographika has a ticket #6463 that is documenting some exceptions we'll need to address. Perhaps you could add this there so it's all in one spot.

* Refactor validation of numeric content read from the mapfile. * Updated create image function with guard to ensure positive image size (matches Cairo approach).

gu-LGB · 2024-09-04T15:18:47Z

mapserver.h

  /************************************************************************/

+  #define MS_LEGEND_KEYSIZE_MIN 5
+  #define MS_LEGEND_KEYSIZE_MAX 200


I'm currently migrating from MS 7.6 to 8.0 and I have a problem with this limitation: So far I could provide legends consisting of a PNG like this:

LEGEND KEYSIZE 216 216 KEYSPACING 7 7 STATUS ON END CLASS KEYIMAGE "./inspire_lbgr/symbol/legende-sorption_matrix.png" NAME " " END

Is there anything against raising the limit to 1000?

sdlime added 27 commits March 30, 2021 22:08

Create coverity-scan.yml

acb708a

Merge branch 'main' of https://github.com/sdlime/mapserver into main

365120a

Update coverity-scan.yml

e0d1d43

Avoid resource leak... (CID 1503409)

7d261af

Merge remote-tracking branch 'upstream/main' into main

7cc1f00

Revert "Avoid resource leak... (CID 1503409)"

e59d3cb

This reverts commit 7d261af.

Updated...

ebb3cd3

Limit action to MapServer/MapServer repo, run every Sunday (for now).

a4e6858

Merge remote-tracking branch 'upstream/main' into main

dac5cdc

Merge remote-tracking branch 'upstream/main' into main

3111bef

Merge remote-tracking branch 'upstream/main' into main

2fc8d75

Merge branch 'main' of https://github.com/sdlime/mapserver into main

db9eb79

Merge remote-tracking branch 'upstream/main' into main

239aa94

Merge remote-tracking branch 'upstream/main' into main

ffca109

Merge remote-tracking branch 'upstream/main' into main

b99e3ea

Merge remote-tracking branch 'upstream/main' into main

6927be2

Merge remote-tracking branch 'upstream/main' into main

37f39d4

Merge remote-tracking branch 'upstream/main' into main

f10e7ea

Merge remote-tracking branch 'upstream/main' into main

7ed9265

Merge remote-tracking branch 'upstream/main' into main

85e2ba9

Merge remote-tracking branch 'upstream/main' into main

830865e

Merge remote-tracking branch 'upstream/main' into main

e42cd06

Merge remote-tracking branch 'upstream/main' into main

5c09ecd

Merge remote-tracking branch 'upstream/main' into main

450daae

Refactor validation of numeric content read from the mapfile.

0267bb5

Updated create image function with guard to ensure positive image siz…

4e70a0d

…e (matches Cairo approach).

Refactored a bit to avoid cppcheck unitialized variable error(s).

28ed629

geographika reviewed Jan 24, 2022

View reviewed changes

sdlime requested a review from rouault January 25, 2022 17:38

sdlime requested review from dmorissette and jmckenna January 25, 2022 17:38

rouault reviewed Jan 25, 2022

View reviewed changes

mapfile.c Show resolved Hide resolved

Pulled number checking guts into a separate function for re-use. Adde…

15df3b8

…d clarifying comments in loadColor(). Resolved a couple of TODO comments.

rouault self-requested a review January 26, 2022 13:46

rouault approved these changes Jan 26, 2022

View reviewed changes

sdlime merged commit e74a176 into MapServer:main Jan 26, 2022

jmckenna added this to the 8.0 Release milestone Jan 26, 2022

sdlime deleted the inv branch January 26, 2022 14:21

This was referenced Jan 26, 2022

Document new Mapfile validation in #6458 MapServer/MapServer-documentation#663

Merged

New Mapfile Validation Rules #6463

Closed

geographika mentioned this pull request Feb 1, 2022

Update schema based on new Mapfile validation rules geographika/mappyfile#138

Merged

gu-LGB reviewed Sep 4, 2024

View reviewed changes

Uh oh!

Conversation

sdlime commented Jan 24, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdlime Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geographika commented Jan 24, 2022

Uh oh!

sdlime commented Jan 25, 2022

Uh oh!

Uh oh!

sdlime commented Jan 26, 2022

Uh oh!

sdlime commented Jan 26, 2022

Uh oh!

geographika commented Jan 26, 2022

Uh oh!

sdlime commented Jan 26, 2022

Uh oh!

jmckenna commented Jan 26, 2022

Uh oh!

ejn commented Jan 26, 2022

Uh oh!

jmckenna commented Jan 26, 2022

Uh oh!

jmckenna commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdlime commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmckenna commented Jan 26, 2022

Uh oh!

jmckenna commented Jan 27, 2022

Uh oh!

jmckenna commented Jan 27, 2022

Uh oh!

sdlime commented Jan 27, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sdlime Jan 24, 2022 •

edited

Loading

jmckenna commented Jan 26, 2022 •

edited

Loading

sdlime commented Jan 26, 2022 •

edited

Loading