Raster WMS-T with OGR tileindex not implementing filter translation support correctly?

[ruhri]

According to
http://osgeo-org.1560.x6.nabble.com/Raster-WMS-T-with-OGR-tileindex-not-implementing-filter-translation-support-td5225415.html
I get similar logs.

I use "Spatialite" as OGR datasource for my time-/tileindex. The service works well and also the time values ar properly evaluated. But for each request I get an errorlog:
LayerDefaultTranslateFilter(): General error message. This data driver does not implement filter translation support.

Maybe this relates to
http://mapserver.org/development/rfc/ms-rfc-91.html
and the way, how the wms_time-logic evaluates time-values from ogr-datasources?

[sdlime]

What does the layer definition look like?

[ruhri]

# DOP-Layer
LAYER
    NAME "dop"
    TILEINDEX "dop_metadaten"
    TILEITEM "pfad"
    METADATA
        "ows_title"                             "DOP"
        "ows_abstract"                          "Ebene für Digitale Orthophotos"
        "ows_srs"                               "EPSG:25832 EPSG:25833 EPSG:3857 EPSG:4258 EPSG:4326 EPSG:31466 EPSG:31467 EPSG:31468 EPSG:31469 EPSG:900913"
        "ows_extent"                            "309000.0 5672000.0 434000.0 5746000.0"
        "wms_timeitem"                          "darstellungsjahr"
        "wms_timeformat"                        "YYYY"
        "wms_timeextent"                        "2006/2015/P1Y"
        "wms_timedefault"                       "2015"
        "wms_opaque"                            "1"
        "wms_metadataurl_format"            "application/vnd.iso.19139+xml"
        "wms_metadataurl_type"              "TC211"
        "wms_metadataurl_href"              "https://geonetzwerk.metropoleruhr.de/geonetwork/srv/ger/csw?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&id=dc5f5a17cb2e0744812ba91b45fd5637c894403b"
    END

    TYPE RASTER

    PROJECTION
        "init=epsg:25832"
    END

    CLASS
        NAME "Orthophoto"
        KEYIMAGE "./dop_keyimage.png"
    END

    MAXSCALEDENOM 3000000
    MINSCALEDENOM 1
END

# Metadaten-Layer
LAYER
    NAME "dop_metadaten"
    CONNECTIONTYPE OGR
    CONNECTION "/data2/dop/DB/DOP.sqlite"
    PROCESSING "CLOSE_CONNECTION=DEFER"
    DATA "SELECT geometrie, pfad, darstellungsjahr, bildflugjahr, gemeinde, kreis, regierungsbezirk, strftime('%d.%m.%Y', bildflugdatum) AS bildflugdatum, bodenaufloesung, strftime('%d.%m.%Y', importdatum) AS importdatum, darstellungsaufloesung FROM vw_bildfluege"
    METADATA
        "ows_title"                             "DOP - Metadaten"
        "ows_abstract"                          "Ebene für Digitale Orthophotos - Metadaten"
        "ows_srs"                               "EPSG:25832 EPSG:25833 EPSG:3857 EPSG:4258 EPSG:4326 EPSG:31466 EPSG:31467 EPSG:31468 EPSG:31469 EPSG:900913"
        "ows_extent"                            "309000.0 5672000.0 434000.0 5746000.0"
        "wms_timeitem"                          "darstellungsjahr"
        "wms_timeformat"                        "YYYY"
        "wms_timeextent"                        "2006/2015/P1Y"
        "wms_timedefault"                       "2015"
        "gml_include_items"                     "all"
        "gml_exclude_items"                     "pfad"
        "gml_geometries"                        "geom"
        "gml_geom_type"                         "polygon"
        "wms_metadataurl_format"            "application/vnd.iso.19139+xml"
        "wms_metadataurl_type"              "TC211"
        "wms_metadataurl_href"              "https://geonetzwerk.metropoleruhr.de/geonetwork/srv/ger/csw?request=GetRecordById&service=CSW&version=2.0.2&elementSetName=full&id=3eecdb33792a1f5d39c6380a1ed64300dc537def"
    END

    HEADER "./html/query-header.html"
    TEMPLATE "./html/query.html"
    FOOTER "./html/query-footer.html"

    TYPE POLYGON

    PROJECTION
        "init=epsg:25832"
    END

    LABELMAXSCALEDENOM 400000

    CLASS
        NAME "Stadtgrenze"
        STYLE
            OUTLINECOLOR 255 255 0
        END
        TEXT ('[gemeinde]' + '|' + '[bildflugjahr]')
        LABEL
            TYPE TRUETYPE
            FONT DejaVuSans-Bold
            SIZE 9
            MINFEATURESIZE AUTO
            ALIGN center
            WRAP "|"
            COLOR 255 255 0
            POSITION CC
            PARTIALS FALSE
        END
    END
END

[ruhri]

I hope this helps, the datasource is a View in Spatialite.
To make it clear: the service works well - also time support - but i get that much logs...
...and debug is set to "DEBUG 0"

[sdlime]

Ah, good. I didn't see anything in the layer definitions that worried me. I'll see if I can make the warning more relevant - only throw it if there is a real need.

[tbonfort]

@sdlime: when is it safe to ignore this warning ?

[ruhri]

Having build mapserver from latest master two days ago, the above posted warning became true, mapserver doesn't filter against time-value from getmap-request anymore.
From my Point of view this explicitely happens with spatialite-datasource as tileindex. Converting this tileindex to shape, it seems to work well.

original error message:
LayerDefaultTranslateFilter(): General error message. This data driver does not implement filter translation support.

[ruhri]

@rouault
I don't know whether your read my post, but I suppose, your work, at least partially, relates to my Problem (--> Fix support for Spatialite views - b5cd66e)

I'm using a SpatialiteView as Tile- and Timeindex for a WMS with aerial images and time Support.
Since a few months ago everything worked well with that service, after some rebuilds in january from master-branch I got the originally posted debugmessage (see above), but everything worked still well.
After a rebuild a few days ago, the Service got very slow. Some further Investigation showed me that timeitem-field was not evaluated anymore, so that mapserver rendered every Image from time series.
Since building mapserver again today (with the hope, that your fix from yesterday would also fix my problem) I got a non working Service with following error message:
--> msDrawMap(): Image handling error. Failed to draw layer named 'dop'. msOGRFileNextShape(): OGR error. Pointer 'hLayer' is NULL in 'OGR_L_GetGeometryColumn'.

A workaround solution for me is to let mapserver not "look" directly in the spatialite-DB (as in my mapfile post above), but establish the db-connection via OGR/VRT. The only difference is the move of the SQL-Statement from mapfile to the vrt-file. Then mapserver does exactly what it should do - filtering against timeitem-field with time-parameter from getmaprequest.

--> I argue, that there's something wrong/different with the spatialite implementation in mapserver. From my point of view it behaves different than generic OGR. I'm using latest GDAL/OGR 2.1.1

[ruhri]

Another hint/Approach: doing the table-join directly in mapfile DATA-element (which was done in the spatialite-View) this also works well, but getting the log-entry:
msOGRFileWhichShapes(): unsetting native_string

[rouault]

It would help if you could provide a

• minimalistic mapfile
• minimalistic spatialite DB + pointed raster
• WMS-T request
  that demonstrate the issue.

The "msOGRFileWhichShapes(): unsetting native_string" warning may appear if the view is not registered as a spatial view. This should normally not result in a misbehaviour (except potential bugs in the implementation...), but in client-side only filtering.

[ruhri]

Hi,
the Spatialite-DB is attached

Sincerely /
Mit freundlichen Grüßen
Im Auftrag

Benjamin Schepers

Luftbild und Geoinformationssysteme
Kronprinzenstraße 35
45128 Essen
Fon: +49 201 2069-232
Fax: +49 201 2069-500
schepers@rvr-online.de

[cid:image001.jpg@01D1E97E.D3A593D0]

Die Regionaldirektorin
Kronprinzenstraße 35
45128 Essen
Zentrale: +49 (0) 201 2069-0
Fax: +49 (0) 201 2069-500
www.metropoleruhr.de

Postfach 10 32 64
45032 Essen

Steuernummer: RVR 112/5797/0116
USt.-ldNr.: DE 173867500

Diese E-Mail koennte vertrauliche und/oder rechtlich geschuetzte Informationen enthalten.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail sind nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Von: Even Rouault [mailto:notifications@github.com]
Gesendet: Donnerstag, 28. Juli 2016 18:50
An: mapserver/mapserver
Cc: Schepers, Benjamin; Author
Betreff: Re: [mapserver/mapserver] Raster WMS-T with OGR tileindex not implementing filter translation support correctly? (#5218)

It would help if you could provide a

• minimalistic mapfile
• minimalistic spatialite DB + pointed raster
• WMS-T request that demonstrate the issue.

The "msOGRFileWhichShapes(): unsetting native_string" warning may appear if the view is not registered as a spatial view. This should normally not result in a misbehaviour (except potential bugs in the implementation...), but in client-side only filtering.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/5218#issuecomment-235954980, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFhgBxIwwwlprnWtxwQGhuYNzw7tSymhks5qaN29gaJpZM4G-spP.

[ruhri]

Hi, thanks for your Response.
can't send you the raster-data (huge bigtiffs... a few TB), just the Spatialite-DB.

The Spatial View was build in QGIS DB-Manager SQL-Window - it's just a simple join of two tables, one with geom and some string/time-formatting operations:
CREATE VIEW vw_bildfluege AS
SELECT
b.pkid AS rowid,
b.pkid AS bildflug_id,
v.geometrie AS geometrie,
v.gemeindekennzahl AS gemeindekennzahl,
v.regierungsbezirk AS regierungsbezirk,
v.kreis AS kreis,
v.gemeinde AS gemeinde,
v.gemeinde_ohne_umlaut AS gemeinde_ohne_umlaut,
strftime('%d.%m.%Y', b.bildflugdatum) AS bildflugdatum,
strftime('%d.%m.%Y', b.importdatum) AS importdatum,
strftime('%Y', bildflugdatum) AS bildflugjahr,
b.darstellungsjahr AS darstellungsjahr,
b.bodenaufloesung AS bodenaufloesung,
b.darstellungsaufloesung AS darstellungsaufloesung,
'/data2/dop/'||strftime('%Y', bildflugdatum)||'/'||gemeinde_ohne_umlaut||'.tif' AS pfad
FROM bildflug AS b
JOIN verwaltung AS v ON (b.verwaltungs_id = v.pkid);

INSERT INTO views_geometry_columns
(view_name, view_geometry, view_rowid, f_table_name, f_geometry_column, read_only)
VALUES ('vw_bildfluege', 'geometrie', 'rowid', 'verwaltung', 'geometrie', 0);

It works well in QGIS and did in mapserver 6.4, Problems began with mapserver 7 this year.
The original Mapfile was posted above, I build a test-Scenario with two more test-files, exactly the same but different DATA-String:

1. (spatialite-view) :
   DATA "SELECT geometrie, pfad, darstellungsjahr, bildflugjahr, gemeinde, kreis, regierungsbezirk, bildflugdatum, bodenaufloesung, importdatum, darstellungsaufloesung FROM vw_bildfluege"
2. (JOIN in mapfile - same as in view) :
   DATA "SELECT v.geometrie, '/data2/dop/'||strftime('%Y', b.bildflugdatum)||'/'||v.gemeinde_ohne_umlaut||'.tif' AS pfad, b.darstellungsjahr, strftime('%Y', b.bildflugdatum) AS bildflugjahr, v.gemeinde, v.kreis, v.regierungsbezirk, strftime('%d.%m.%Y', b.bildflugdatum) AS bildflugdatum, b.bodenaufloesung, strftime('%d.%m.%Y', b.importdatum) AS importdatum, b.darstellungsaufloesung FROM bildflug AS b JOIN verwaltung AS v ON (b.verwaltungs_id = v.pkid)"

The first doesn't work (view) - errorlog:
[Fri Jul 29 09:34:36 2016].720332 CGI Request 3 on process 23274
[Fri Jul 29 09:34:36 2016].721226 msDrawMap(): rendering using outputformat named JPEG (AGG/JPEG).
[Fri Jul 29 09:34:36 2016].721241 msDrawMap(): WMS/WFS set-up and query, 0.000s
[Fri Jul 29 09:34:36 2016].721251 msDrawRasterLayerLow(dop): entering.
[Fri Jul 29 09:34:36 2016].722614 msOGRFileNextShape(): OGR error. Pointer 'hLayer' is NULL in 'OGR_L_GetGeometryColumn'.
[Fri Jul 29 09:34:36 2016].722628 msDrawMap(): Image handling error. Failed to draw layer named 'dop'.
[Fri Jul 29 09:34:36 2016].722642 mapserv request processing time (msLoadMap not incl.): 0.002s
[Fri Jul 29 09:34:36 2016].722646 msFreeMap(): freeing map at 0xd2b71e0.

The second one (mapfile JOIN) works - exactly the same request:
[Fri Jul 29 09:35:56 2016].824662 CGI Request 1 on process 23271
[Fri Jul 29 09:35:56 2016].825544 msDrawMap(): rendering using outputformat named JPEG (AGG/JPEG).
[Fri Jul 29 09:35:56 2016].825561 msDrawMap(): WMS/WFS set-up and query, 0.000s
[Fri Jul 29 09:35:56 2016].825571 msDrawRasterLayerLow(dop): entering.
[Fri Jul 29 09:35:56 2016].830806 msOGRFileWhichShapes(): unsetting native_string
[Fri Jul 29 09:35:57 2016].103029 msDrawMap(): Layer 0 (dop), 0.277s
[Fri Jul 29 09:35:57 2016].103744 msOGRFileWhichShapes(): unsetting native_string
[Fri Jul 29 09:35:57 2016].125761 msDrawMap(): Layer 1 (dop_metadaten), 0.023s
[Fri Jul 29 09:35:57 2016].127122 msDrawMap(): Drawing Label Cache, 0.001s
[Fri Jul 29 09:35:57 2016].127130 msDrawMap() total time: 0.302s
[Fri Jul 29 09:35:57 2016].264118 msSaveImage(stdout) total time: 0.137s
[Fri Jul 29 09:35:57 2016].264171 mapserv request processing time (msLoadMap not incl.): 0.440s
[Fri Jul 29 09:35:57 2016].264182 msFreeMap(): freeing map at 0xeb9720.

the only abnormal thing in log is the warning 'unsetting native_string'

maybe I did something wrong with the view? got it from https://www.gaia-gis.it/gaia-sins/spatialite-cookbook/html/sp-view.html

with my last build from mapserver from April / May (before the huge Spatialite-Code-changes) method one also worked, but with warnings:
LayerDefaultTranslateFilter(): General error message. This data driver does not implement filter translation support.

THX

[rouault]

I didn't get your .db. I've pushed something that might fix your issue.