Skip to content
This repository

Is layer_by_sql in ogr plugin working? #1029

Open
goododd opened this Issue · 16 comments

4 participants

goododd Dane Springmeyer Joel Brown Wilhelm Berg
goododd

Sorry for double posting to the Google group. I had a very small amount of data in ms sql server and was able to render image if I do Parameter name="layer" planet_osm_line (sorry, it should be in xml)

I'd like to create a layer based on a sql query as follows, (xml format avoided as it is not allowed in the edit box)

Datasource
Parameter name="type" ogr
Parameter name="layer_by_sql" select * from planet_osm_line
Parameter name="string" some connection string to mssql server

I tried the same sql using ogrinfo and it works. However, mapnik didn't complain anything but the image is empty. I tried to make a invalid select above to show that the ogr plugin is actually called (with some errors).

Any hints? Thanks.

Dane Springmeyer
Owner

The layer_by_sql is very new. I've personally not had a chance to test it yet. I would advise trying to get it working for some very simple case like a small shapefile.

goododd

My sql profiler shows the sql query reached the server but somehow the returned data is not considered as a layer. No error complained. BTW, I am using the windows binaries RC. So maybe I should wait for the 2.1 release? Thanks.

Dane Springmeyer
Owner

If seen this type of behavior when the geometry type is not proper OGC WKB. Basically features are parsed but geometries are empty. Could this be the case?

Joel Brown

Was there every a resolution to this? I am seeing the same exact behavior in the windows 2.2 and 2.3 sdks (http://mapnik.s3.amazonaws.com/dist/dev/mapnik-v2.3.0.7z).

layer renders as expected with the following mapping xml:

  <Parameter name="type">ogr</Parameter>
  <Parameter name="string">MSSQL:server=.\SQLEXPRESS;database=gis;trusted_connection=yes;</Parameter>
  <Parameter name="layer_by_index">0</Parameter>

But as soon as i use the layer_by_sql parameter the layer fails to render. I can even see the sql statement getting executed on the SQL Server instance just like @goododd :

  <Parameter name="type">ogr</Parameter>
  <Parameter name="string">MSSQL:server=.\SQLEXPRESS;database=gis;trusted_connection=yes;</Parameter>
  <Parameter name="layer_by_sql>SELECT ogr_fid, ogr_geometry FROM dbo.land</Parameter>

dbo.land was imported from a shapefile via ogr2ogr. The shapefile can be successfully rendered by mapnik with the ogr driver. It can even be rendered using the layer_by_sql option:

  <Parameter name="string">D:\TileCook\TileCook.Web\App_Data\Config\land.shp</Parameter>
  <Parameter name="layer_by_sql">SELECT * from land</Parameter>

I suspected it might have something to do with mssql ogr driver but i cant pinpoint what is going on. ogrinfo has no problem reading the mssql features. I tried creating a vrt file to force an OGR layer envelope and feature count:

<OGRVRTLayer name="test">
  <SrcDataSource>MSSQL:server=.\SQLEXPRESS;database=gis;trusted_connection=yes;</SrcDataSource> 
  <SrcSQL>SELECT ogr_fid, ogr_geometry FROM dbo.land</SrcSQL>
  <ExtentXMin>-20037508.342789255</ExtentXMin>
  <ExtentYMin>-20037508.342789255</ExtentYMin>
  <ExtentXMax>20037508.342789255</ExtentXMax>
  <ExtentYMax>20037508.342789255</ExtentYMax>
  <FeatureCount>500</FeatureCount>
</OGRVRTLayer>

Still did not work. Lastly, i tried to troubleshoot things on the mapnik end but did not get very far.

//get mssql layer
    mapnik::layer lay =_map->getLayer(0);
    mapnik::datasource_ptr ds = lay.datasource();
    mapnik::box2d<double> layExt = ds->envelope();
    mapnik::query q(layExt, 1.0);
    mapnik::featureset_ptr fs = ds->features(q);
    mapnik::feature_ptr feat = fs->next();
    while (feat) {feat = fs->next();}

I would expect the while loop to iterate for as many records are returned by the SQL statement. However, the code never enters the loop. One little nugget i noticed is that layExt is (0,0,0,0) unless i force an OGR layer extent using the vrt layer as mentioned above.

I'd really like to get a solution on this. It could be a big win for windows integration. I'm up for debugging this further. I just feel i have hit a wall at the moment.

Wilhelm Berg
Collaborator

@kernelsanders I was able to compile your NET-Mapnik :smile: and I'm also able to reproduce the problem on Windows with SQL Server.
Will take a look at it during the coming week.

Joel Brown

Sounds good. Thanks for jumping on this issue. Glad you were able to compile NET-Mapnik. I was also able to reproduce this issue using tilemill/cartoCSS.

As a workaround, i created a database view that contains the SQL logic. When i reference the view using the "layer" parameter i can successfully render the layer with mapnik. To me, this seems to rule out data issues. I feel like there is something funky going on when OGR hands off an OGRLayer object created via exeuctesql or something like that.

Thanks again!

Dane Springmeyer
Owner
Wilhelm Berg
Collaborator

As far as I can see there is no spatial filter: OGRGeometry* spatial_filter = nullptr;

My findings so far:
I've replicated the plugin code in C# (was easier and faster for me :smirk: ).
layer_by_sql works and gets the requested features, except that the extent reported is 0/0 0/0 so I think the problem is further up the line, when the extent gets evaluated.

ogr-datasource-executesql

Ogr.RegisterAll();
DataSource ds = Ogr.Open( @"MSSQL:server=.\sqlexpress2012;database=netmapnik;trusted_connection=yes;Tables=countries", 0 );
Driver drv = ds.GetDriver();
if (null == drv) {
    Console.WriteLine( "cannot get driver" );
    Environment.Exit( 1 );
}

int lyrCnt = ds.GetLayerCount();
for (int i = 0; i < lyrCnt; i++) {
    Layer lyrByIdx = ds.GetLayerByIndex( i );
    Envelope ext = new Envelope();
    lyrByIdx.GetExtent( ext, 1 );
    FeatureDefn def = lyrByIdx.GetLayerDefn();
    Console.WriteLine( string.Format(
        "---- By Index ----{0}Name:{1}{0}FeatCnt:{2}{0}Ext:{3}/{4} {5}/{6}{0}"
        , Environment.NewLine
        , def.GetName()
        , lyrByIdx.GetFeatureCount( 1 )
        , ext.MinX
        , ext.MinY
        , ext.MaxX
        , ext.MaxY
    ) );
}

Layer lyrBySql = ds.ExecuteSQL( "SELECT ogr_fid, ogr_geometry FROM countries", null, null );
Envelope extBySql = new Envelope();
lyrBySql.GetExtent( extBySql, 1 );
FeatureDefn defBySql = lyrBySql.GetLayerDefn();
Console.WriteLine( string.Format(
    "---- By SQL ----{0}Name:{1}{0}FeatCnt:{2}{0}Ext:{3}/{4} {5}/{6}{0}"
    , Environment.NewLine
    , defBySql.GetName()
    , lyrBySql.GetFeatureCount( 1 )
    , extBySql.MinX
    , extBySql.MinY
    , extBySql.MaxX
    , extBySql.MaxY
) );

Console.WriteLine( "Features by SQL:" );
Feature f;
while (null != (f = lyrBySql.GetNextFeature())) {
    Envelope fExt = new Envelope();
    Geometry geom = f.GetGeometryRef();
    geom.GetEnvelope( fExt );
    Console.WriteLine( string.Format(
        "[{0}]: {1:0.000}/{2:0.000}\t{3:0.000}/{4:0.000}"
        , f.GetFID()
        , fExt.MinX
        , fExt.MinY
        , fExt.MaxX
        , fExt.MaxY
    ) );
}
Wilhelm Berg
Collaborator

It looks like, this is not implemented in SQL Server:
"SQL Server Spatial follows OGC closely and therefore does not implement this functionality. Neither it has implemented it as a so-called extended geometry method."
http://barendgehrels.blogspot.co.at/2011/04/extent-of-sql-server-spatial-table.html

Joel Brown

GetExtent with SELECT statement is not implemented in OGR:

This was my finding as well. That is why i tried using vrt layer which allows you to explicitly define an extent:

<OGRVRTLayer name="test">
  <SrcDataSource>MSSQL:server=.\SQLEXPRESS;database=gis;trusted_connection=yes;</SrcDataSource> 
  <SrcSQL>SELECT ogr_fid, ogr_geometry FROM dbo.land</SrcSQL>
  <ExtentXMin>-20037508.342789255</ExtentXMin>
  <ExtentYMin>-20037508.342789255</ExtentYMin>
  <ExtentXMax>20037508.342789255</ExtentXMax>
  <ExtentYMax>20037508.342789255</ExtentYMax>
  <FeatureCount>500</FeatureCount>
</OGRVRTLayer>

The extent option is only available at GDAL >= v1.10. I had to pull down mapnik v2.3.0 to test this feature since v2.2.0 uses an older GDAL version i think. When i run the mapnik code below with the vrt layer i successfully get an extent back from "ds->envelope()" call. This is where i got tripped up. It appears that mapnik is successfully reading an extent from the OGRLayer object. However, I still can't get features to render.

//get mssql layer
    mapnik::layer lay =_map->getLayer(0);
    mapnik::datasource_ptr ds = lay.datasource();
    mapnik::box2d<double> layExt = ds->envelope();
    mapnik::query q(layExt, 1.0);
    mapnik::featureset_ptr fs = ds->features(q);
    mapnik::feature_ptr feat = fs->next();
    while (feat) {feat = fs->next();}
Dane Springmeyer
Owner

Great findings. I've created #2260 to track fixing the handling of failed extents (this will avoid needing the VRT workaround). @BergWerkGIS - interesting that spatial filters are not implemented too. What I wonders is if commenting these lines might fix things: https://github.com/mapnik/mapnik/blob/cbdd112223b42576da403db9f19dc984217ea331/plugins/input/ogr/ogr_featureset.cpp#L58-L61.

@kernelsanders btw, would you be interested in a native MSSQL Mapnik plugin? Others have contacted me requesting this and I'm looking for development help to get it done.

Wilhelm Berg
Collaborator

@springmeyer don't know if commenting these lines will help. ATM I don't have an environment to debug mapnik (maybe this will be easier with the gyp version)?
Probably checking the extent after getting the data (layer_by_sql) and setting it manually by iterating the features, if there are any, might be a temporary workaround till we find a solution with better performance.

Joel Brown

@kernelsanders btw, would you be interested in a native MSSQL Mapnik plugin? Others have contacted me requesting this and I'm looking for development help to get it done.

Yes, a MSSQL plugin would be awesome. What kind of resources do you need to get this done?

Dane Springmeyer
Owner

What kind of resources do you need to get this done?

A developer with the ambition to tackle it. I don't have time, but I could lightly advise someone on how Mapnik datasource plugins work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.