Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows build #81

Closed
chrislukic opened this Issue Sep 15, 2015 · 34 comments

Comments

Projects
None yet
@chrislukic
Copy link

chrislukic commented Sep 15, 2015

Any thoughts on creating package for Windows? Or perhaps adding instructions for creating a windows build to the documentation?

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Sep 15, 2015

I hope I can make it work on Windows, but I don't think it's going to be easy, since it relies heavily on memory-mapped files. Maybe there's a compatibility layer somewhere that will provide an mmap-like interface in Windows.

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented Sep 15, 2015

That was my worry. I'm trying to implement a tippecanoe/Mapbox API upload workflow on a windows server. It would make things much easier to keep it all in the same environment.

Admittedly out of my depth, but some Google research seems to suggest MapViewOfFile might be of use.

@BergWerkGIS

This comment has been minimized.

Copy link
Member

BergWerkGIS commented Sep 19, 2015

There seems to be a Windows port of mmap (https://code.google.com/p/mman-win32/source/browse/#svn%2Ftrunk), but it looks abandoned (files are 3 years old).

I took a quick stab and tried to get tippecanoe compile on Windows.
https://github.com/mapbox/tippecanoe/compare/master...BergWerkGIS:master?diff=split&name=master

I was not successfull and don't have the bandwidth to pursue, as there are also some other code changes necessary.

e.g. variable length arrays are not supported by Visual Studio:

char *metakey[nprop];
char *metaval[nprop];
int metatype[nprop];

tippecanoe/geojson.c

Lines 454 to 456 in 18647d1

char *metakey[nprop];
char *metaval[nprop];
int metatype[nprop];

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Sep 22, 2015

Thanks for trying to make it work!

I could probably work around the variable-length arrays just by dragging more of the code onto the C++ side and using vectors instead of arrays, and will prioritize that if it will help you.

@BergWerkGIS

This comment has been minimized.

Copy link
Member

BergWerkGIS commented Sep 23, 2015

@ericfischer

dragging more of the code onto the C++ side and using vectors instead of arrays, and will prioritize that if it will help you.

Cool, would be really great to have tippecanoe on Windows, too.
Could you ping me after you have refactored, that I can give it another try.

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented Oct 2, 2015

Any further thoughts on this? I'm approaching a crossroads soon, and debating whether to start looking into adding a linux server for the Tippecanoe workflow. I'd definitely like to avoid this, but don't want to pin my hopes on a low probability outcome.

@rowanwins

This comment has been minimized.

Copy link

rowanwins commented Dec 16, 2015

Gday all,

Just throwing in a quick comment here although I know next to nothing about memory mapped files or C++...

It looks like the boost library provides a cross-platform solution to memory mapped files.

Someone has thrown together some sample code here which might show a bit of a starting point.

Wish I could help more because I'd love to see tippecanoe working on windows machines :)

Cheers
Rowan

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Dec 17, 2015

Sorry to have missed your message on Oct 2, @chrislukic!

I'll have to try the boost abstraction, @rowanwins. Thanks for telling me about it! I hope I'll have a chance to try this, but I have absolutely no Windows development experience so I hope no one is counting on having this be available any time soon.

@rowanwins

This comment has been minimized.

Copy link

rowanwins commented Dec 18, 2015

no rush here @ericfischer , give me a yell if you need someone to do some testing/attempted debugging!

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Mar 30, 2016

Sounds like Linux command line tools may soon work unchanged on Windows.

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented Mar 30, 2016

Such a strange and wonderful thing.

@jingsam

This comment has been minimized.

Copy link
Contributor

jingsam commented Apr 1, 2016

Use Cygwin. Tippecanoe works unmodified on Cygwin.

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Apr 1, 2016

Oh fantastic, that's great to hear @jingsam!

@BergWerkGIS

This comment has been minimized.

Copy link
Member

BergWerkGIS commented May 20, 2016

Thanks, @jingsam for verifying that Cygwin is a workaround.
Closing for now, maybe revisit when Bash on Windows is generally available.

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented May 23, 2016

@jingsam First time Cygwin user, so any help is much appreciated. Installed devel zlib, sqllite, make, g++, packages, but still get this error from the make.

image

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented May 23, 2016

@jingsam Was eventually able to get it to build. Added the flag -U__STRICT_ANSI__ to the make file. thx.

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented May 23, 2016

Thanks @chrislukic. I'm glad you were able to get it working!

@BergWerkGIS

This comment has been minimized.

Copy link
Member

BergWerkGIS commented May 27, 2016

Sounds like Linux command line tools may soon work unchanged on Windows.

YES!
Just updated to the latest Windows Insider Preview Build of Windows 10 and installed Ubuntu Bash on Windows.

Built a tippecanoe release build (with clang@3.5) without any modifications and make test finished without any errors.


Remember: the resulting binaries are native ELF binaries and not Windows binaries.

bergw@MB:~/tippecanoe$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty
bergw@MB:~/tippecanoe$
@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented May 27, 2016

Thanks for checking on this, @BergWerkGIS! Glad to hear it works.

@sainsb

This comment has been minimized.

Copy link

sainsb commented Nov 16, 2016

For those ultimate cpp noobs - append the -U__STRICT_ANSI__ to line 10 CXXFLAGS of the makefile.

Was able to build in Cygwin on windows after applying @chrislukic solution

@lihuadong1988

This comment has been minimized.

Copy link

lihuadong1988 commented Mar 8, 2017

@chrislukic tippecanoe works very slow if json file is too big,have you ever met this question?

@chrislukic

This comment has been minimized.

Copy link
Author

chrislukic commented Mar 8, 2017

@lihuadong1988 Unfortunately yes. Performance was a big enough issue, we ended up abandoning cygwin and moving the processing to AWS.

@pathmapper

This comment has been minimized.

Copy link

pathmapper commented Jan 23, 2018

Just found this step-by-step guide for Tippecanoe with Cygwin and tried it successfully on Windows 7:
https://github.com/GISupportICRC/ArcGIS2Mapbox#installing-tippecanoe-on-windows

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Jan 23, 2018

Thank you @pathmapper! I'll add that link to the documentation.

@jingsam

This comment has been minimized.

Copy link
Contributor

jingsam commented Jan 24, 2018

I have made two days efforts trying to make tippecanoe work on Windows. There are 4 ways:

  1. Use Ubuntu Bash on Windows. Tippecanoe works perfectly on WSL. I did not test the performance on this environment. The problem is that Ubuntu Bash on Windows is only available on Windows 10 or the latest Windows Server. So the way is not working on Windows 7 or XP.

  2. Use Cygwin. Cygwin provides a POSIX environment on Windows. Tippecanoe can be compile with -U__STRICT_ANSI__ flag in Cygwin. This way works on any version of Windows. The problem is that the performance is very low when you have to process large geojson files.

  3. Use VirtualBox or Docker. You can install a linux environment in VirtualBox or Docker. The way also works on any versions of Windows. The problem is that it seems too heavy to install tippecanoe on Windows. The performance is not tested. I guess it will be even lower than Cygwin.

  4. Use MinGW-w64. I am trying to compile a native tippecanoe with MinGW and not yet succeed. Tippecanoe is POSIX compatible, not cross-platform. It heavily relys on some POSIX headers as follows:

#include <sys/time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/sysctl.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/param.h>
#include <sys/mount.h>
#include <sys/statfs.h>
#include <sys/wait.h>

MinGW provides some POSIX headers, but most of them is not available. For now I think it is not possible to make a native tippecanoe on windows using MinGW.

Option 4 is the only possible way to make tippecanoe works efficiently on any versions of Windows. However, Option 4 is not possible for now.

The fact that Tippecanoe can not works natively on Windows is indeed a pain for me. My use case is that our data processing environment are all Windows 7 and offline. The data we need to process is very large, from hundreds of GBs to TBs. So we urge to find high performance vector tiles generating tools.

So, Could you give us some suggestions or tips to make tippecanoe work natively on Windows. Are these POSIX features vital to tippecanoe? Can we find alternatives of these features on Windows? We want to help to make it work. If you provide some tips, maybe the processing would be a little easier.

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented Jan 24, 2018

Thanks for digging into this @jingsam.

Yes, Tippecanoe depends heavily on memory-mapped I/O (<sys/mman.h>), so it is not possible to implement strictly within Standard C without any operating system extensions. That said, Windows does have APIs for memory-mapped I/O. I just don't know anything about them myself.

The one extra-Unixy thing that Tippecanoe does is to open temporary files and then immediately delete them so that they are anonymous. I don't think that works on Windows, but Tippecanoe could be changed to delete its temporary files after it is done with them instead.

Going through these headers in order:

  • sys/time.h is probably actually not necessary
  • sys/types.h is needed for creating POSIX threads
  • sys/stat.h is needed to find the sizes of files. Could be replaced by a Windows API
  • sys/sysctl.h is needed to find the size of main memory on MacOS. Could be replaced by a Windows API.
  • sys/mman.h is needed to memory-map files. Could be replaced by a Windows API
  • sys/resource.h is needed to check how many files can be open at once. Could be replaced by a Windows API.
  • sys/param.h is MacOS-only and may not actually be needed
  • sys/mount.h is needed to check how much disk space is available on MacOS. Could be replaced by a Windows API or stubbed out.
  • sys/statfs.h is needed to check how much disk space is available on Linux. Could be replaced by a Windows API or stubbed out.
  • sys/wait.h is needed to start child processes for shell filters. This mechanism would probably need to be removed on Windows.
@medv

This comment has been minimized.

@skaletech

This comment has been minimized.

Copy link

skaletech commented May 2, 2018

So, there are some libraries already ported to windows:

mman-win32
zlib is on nuget
pthreads is on nuget

Alot of the rest could be resolved with libgwin32c, but I am not a c/c++ expert anymore.

It would be really slick to be able to run this on windows.

@bblacksh

This comment has been minimized.

Copy link

bblacksh commented May 3, 2018

I get an error with vasprintf when i try to compile using Cygwin, does anyone have a workaround for this?

g++ -MMD -I/usr/local/include -I. -g -Wall -Wshadow -Wsign-compare -Wextra -Wun reachable-code -Wuninitialized -Wshadow -O3 -DNDEBUG -std=c++11 -U__STRICT_ANSI __ -c -o write_json.o write_json.cpp write_json.cpp: In member function ‘void json_writer::aprintf(const char*, ...)’ : write_json.cpp:205:32: error: ‘vasprintf’ was not declared in this scope if (vasprintf(&tmp, format, ap) < 0) { ^ make: *** [Makefile:74: write_json.o] Error 1

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented May 4, 2018

@bblacksh I think I probably need to add

#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif

to the top of write_json.cpp so Cygwin will get the declaration of vasprintf. If this works for you, please let me know and I'll add it.

@bblacksh

This comment has been minimized.

Copy link

bblacksh commented May 4, 2018

@jmasselink

This comment has been minimized.

Copy link

jmasselink commented May 15, 2018

I experienced the error with vasprintf when compiling with CygWin but in the tile-join.cpp file.
Added the def to the top of tile-join.cpp and this error disappeared.

#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented May 15, 2018

Thanks @jmasselink. Tile-join also uses vasprintf so I'll add the #define there.

@ericfischer

This comment has been minimized.

Copy link
Contributor

ericfischer commented May 15, 2018

This will be in #569

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.