Building with MrSid support enabled

[weagle08]

Sorry for being a complete noob and also for posting this as an issue but I didn't see anywhere else to post. My question is how to build this with MrSid support enabled on Windows?

[brianreavis]

Because it's proprietary, we don't have a great way to bundle it by default. I would suggest building GDAL with support for MrSid on your system (tutorial), then use the --build-from-source --shared_gdal flags when installing node-gdal.

[weagle08]

Ok, so I have gdal showing MrSid as a supported format when i do gdalinfo --formats, and I ended up having to pull the node-gdal code down and building it in VS but what do I do once I've got it built in visual studio? And just because I build the node-gdal code with VS doesn't mean I have MrSid support so how do I build with MSBuild or VS so that I can use the --shared_gdal flag?

[weagle08]

I'm new to the C++ side of node modules. Also when I used npm install gdal --build-from-source --shared_gdal I got a lot of linker errors:

typed_array.obj : error LNK2001: unresolved external symbol GDALGetDataTypeName [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "public: __cdecl GDALMultiDomainMetadata::~GDALMultiDomainMetadata(void)" (??1GDALMultiDomainMetadata@@qeaa@XZ) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol CSLFindString [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "public: virtual __cdecl GDALNoDataMaskBand::~GDALNoDataMaskBand(void)" (??1GDALNoDataMaskBand@@UEAA@XZ) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "protected: virtual enum CPLErr __cdecl GDALNoDataMaskBand::IRasterIO(enum GDALRWFlag,int,int,int,int,void *,int,int,enum GDALDataType,int,int)" (?IRasterIO@GDALNoDataMaskBand@@Meaa?AW4CPLErr@@W4GDALRWFlag@@HHHHPEAXHHW4GDALDataType@@hh@Z) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "protected: virtual enum CPLErr __cdecl GDALNoDataMaskBand::IReadBlock(int,int,void *)" (?IReadBlock@GDALNoDataMaskBand@@Meaa?AW4CPLErr@@HHPEAX@Z) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "public: virtual __cdecl GDALNoDataValuesMaskBand::~GDALNoDataValuesMaskBand(void)" (??1GDALNoDataValuesMaskBand@@UEAA@XZ) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
typed_array.obj : error LNK2001: unresolved external symbol "protected: virtual enum CPLErr __cdecl GDALNoDataValuesMaskBand::IReadBlock(int,int,void *)" (?IReadBlock@GDALNoDataValuesMaskBand@@Meaa?AW4CPLErr@@HHPEAX@Z) [S:\dev\Microcosm\node_modules\gdal\build\gdal.vcxproj]
.
.
.

[springmeyer]

@BergWerkGIS and I have both tried to build with --shared_gdal but hit problems. I think at this point --shared_gdal only works predictably against a libgdal that has actually been compiled by node-gdal (except it does seem to work again apt-packaged gdal on linux). To state that another way: I for --shared_gdal to work well for people's custom/source compiled gdal I think we'll need to do some more checking to dynamically determine how the shared gdal is built, what version it is, and adapt to it in custom ways. This is sort of a large task: so I've been meaning to dig into it, but honestly not getting around to it for fear it's a bit of a rabbit hole.

So, another alternative would be to try supporting mrsid and other types or proprietary drivers as plugins. Basically people would build the plugin using the same version of gdal that node-gdal bundles (1.11.1) and then set GDAL_DRIVER_PATH to try to get it the plugin registered. I've not tested this, but it should work :)

[hh]

heh, careful with the @hh mentions ;)

[wilhelmberg]

@springmeyer

👍 to proprietary drivers as plugins.
But nevertheless, I think, even in this case, we have to be careful about runtime (vc120, vc140) and /MT/MD.

@brianreavis @springmeyer

How about we change the build scripts to build with MrSID support, if it is there, e.g. in deps/mrsid and without, if it's not there.
So it's the user's duty to download and put into the right place.
Would that be a feasible workaround concerning the license?

@weagle08

I was never able to get --shared_gdal working with a custom gdal package that did not stem from building node-gdal itself.

It seems that not just the gdal version is a problem, but also the way how gdal is built:
e.g.: common.gypi vs makefile.vc

The orignal intention of --shared_gdal was to reduce build times by not building gdal every time node-gdal was built.

So the workflow was:

• 1st build (slow): build node-gdal fully, which created the shared gdal package
• next builds (fast): build just node-gdal, relying on the shared package

[weagle08]

@springmeyer @BergWerkGIS @brianreavis Ok, so I finally had time to try springmeyer's suggestion and it appears that it does try to load the plugins from my GDAL_DRIVER_PATH, however my MrSID plugin had been built with a different version of the vc compiler so I tried to download the MrSID SDK from lizardtech and build it but when I built it with gdal I was getting an error about no suitable MrSID sdk version so I'll have to chase this issue down more tomorrow. But just wanted to let you know that your suggestion appears that it will work. Will keep you updated. Built using MrSID_DSDK_9.1.0.4045_win64_vc12 version.
Just fyi, I was just looking into this because at work we are using a popular WMS server and it has been giving us issues (there is no reason a WMS server has to be so tedious and I don't like java anyway) and I was toying with the idea of creating a WMS server using nodejs and perhaps mapnik as the tiler when I stumbled upon this project.. Nice work and I'll try to help contribute if I can, and excuse my noobish questions... just learning here!

[springmeyer]

@weagle08 thanks for the update and look forward to hearing progress. As far as wms/node.js/mapnik, if you have not seen https://github.com/mapbox/landspeed.js, check it out. It does not implement the entire spec but it might be a good start for what you need. Bug @mattmakesmaps if you need GetCaps support as I think he has a fork that added that.

[weagle08]

@springmeyer no I had not seen landspeed... very cool, I'll have to check it out!

I have an issue maybe you guys can help me with... I can build GDAL just fine, but when I go to build the MrSID plugin I cannot get past this error:

S:\gdal\gdal-1.11.1\frmts\mrsid>nmake /f makefile.vc plugin

Microsoft (R) Program Maintenance Utility Version 12.00.21005.1
Copyright (C) Microsoft Corporation. All rights reserved.

nmake.opt(455) : fatal error U1050: No suitable MrSID Raster DLL was found in S:
\gdal\MrSID_DSDK_9.1.0.4045_win64_vc12\Raster_DSDK\lib.
Stop.

I've tried several different options in the nmake.opt file to no avail any ideas on what I'm missing?

[weagle08]

So to fix the No suitable MrSID Raster DLL... problem you have to go to the /frmts/mrsid/nmake.opt file and update the lines for Find and validate the specific DLL and add it to MRSID_LIB to match your version of the MrSID DSDK.

[weagle08]

@BergWerkGIS @springmeyer @brianreavis If I wanted to use my build of GDAL I've created, how do I go about doing that? How do I make my build the shared package used by the --shared_gdal flag?

[springmeyer]

Yes, if you pass --shared_gdal then node-gdal will not build its internal gdal copy and instead will use an external copy. On windows this triggers some strict assumptions about where that external gdal is. On unix it triggers calling out to gdal-config. See the code block here: https://github.com/naturalatlas/node-gdal/blob/master/binding.gyp#L77-L115.

[wilhelmberg]

The syntax of shared_gdal is like this:

npm install --build-from-source --msvs_version=2013 --shared_gdal=c:/projects/node-gdal/deplibs/

And the directoy tree within c:/projects/node-gdal/deplibs/ is expected to look like this:

But, as I said above:

• This shared_gdal package was created by building node-gdal without shared_gdal first and then using that package for the following builds
• I was never able to get node-gdal to build with a shared_gdal package that was not created by building node-gdal itself.

So, I think you would have to integrate the mrsid build directly into the node-gdal build, like it is done for other dependencies:
https://github.com/naturalatlas/node-gdal/tree/master/deps

[weagle08]

Finally working on getting a fix in for this and am having a problem with the gyp file.. how can i do something like this:

"conditions": [
    ["<(mrsid_base) != ''",{
        "variables": {
            "gdal_format_gyps": [
                "./gyp-formats/mrsid.gyp:libgdal_mrsid_frmt"
            ],
            "gdal_format_defs": [
                "FRMT_mrsid=1"
            ]
        }
    }]
]

where mrsid_base is a variable and I want to make sure it is set and not an empty string. I get the following error:

SyntaxError: invalid syntax while evaluating condition 'S:/gdal/MrSID_DSDK-9.1.0.4045-win64-vc12 != ''' in deps\libgdal\libgdal.gyp at character 2.

[weagle08]

if you could give me access to push my branch up i can push it so that you can see where the issue is.

[brianreavis]

@weagle08 If you create a pull request, it'll build it. I'm rusty with gyp, but try quotation marks around the left hand side, like this: https://github.com/naturalatlas/node-gdal/blob/master/common.gypi#L31

[weagle08]

I'm unable to create a pull request... I get a 403 denied for my username...

[weagle08]

I have it completed and think my solution is satisfactory since we may not be able to actually include the lizard tech assemblies and includes.

[weagle08]

@brianreavis can you give permissions to weagle08@gmail.com?

[brianreavis]

If you can't create a pull request, that's a problem with github and you should contact them. It seems you haven't updated your fork, though? If you really can't create a PR for whatever reason, at least update it on your fork and I'll pull it into a new branch.

[weagle08]

is my solution for building with MrSid support a workable solution for this module? I've issued a pull request and have tested that it works.

[weagle08]

PR #136 would resolve this.