Support building and running on Windows

[xiamx]

Hi,

I added support for building on windows using Visual Studio 2015.

Binary builds can be found here

[ghost]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at cla@fb.com. Thanks!

[ghost]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[xiamx]

Hi devs, any chance of merging this in?

[matanox]

Indeed having a build script or whatever for Windows, for those rare events where one must do such things, would be very nice.

[akcom]

Hello, what is needed to merge this pull request? Is there anything I can do to help? I know this is now a version behind.

[xiamx]

I can sync up this branch with latest on master. I would just like a member from the fasttext team to take a look at it and see if this can be merged

[xiamx]

I've merged upstream changes with mine. Here's a new binary build as of 2017-05-13 https://github.com/xiamx/fastText/releases/tag/v0.20170513

[xcapt]

Hello, has this been merged?

[xiamx]

Nope. But if using an earlier version is fine for you, try https://github.com/xiamx/fastText/releases

[xcapt]

Yes I have been using this great contribution (thank you for that!) but want to test the latest versions as well.

[cpuhrsch]

Hello @xiamx ,

Thank you for your pull request and for being an active member of our community! We recently added support for cmake. This should also work for Windows, but please let me know if this is not the case for you. We also resolved the compilation issue you're mentioning here. I'm going to close this pull request for now, but please feel free to reopen it at any time if your issue isn't resolved.

Thanks,
Christian

[matanox]

Have people experienced smooth Windows installations then? or shall I rather consult the mailing list...

[matanox]

Have the changes for enabling Windows installation been merged then? what is the procedure for building on Windows if so?

[xiamx]

@matanster I have not tried the latest upstream code. But if the project now builds with CMake, it's just about installing CMake on Windows and using the CMake GUI tool to generate a Visual Studio solution file and build using the generated solution file

[matanox]

Brilliant! will try that soon

…

On Sun, Dec 31, 2017 at 6:57 PM, Mengxuan Xia ***@***.***> wrote: @matanster <https://github.com/matanster> I have not tried the latest upstream code. But if the project now builds with CMake, it's just about installing CMake on Windows and using the CMake GUI tool to generate a Visual Studio solution file and build using the generated solution file — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEw746FkMbj9XHGXRXBFpkVtfebUmt1Mks5tF7zigaJpZM4JnhnI> .

[matanox]

By the way, @xiamx do you think it matters which edition of Visual Studio is used? for example we have now three editions to download https://www.visualstudio.com/vs/cplusplus/

[suamin]

@matanster I tried to build the latest code with CMake 3.10.0-rc4 and Visual Studio 2017 (Community), however I had to make few changes to make it work.

• Change line 55 in CMakeLists.txt from install (TARGETS fasttext-shared LIBRARY DESTINATION lib) to install (TARGETS fasttext-shared RUNTIME DESTINATION lib), otherwise configure in CMake will raise CMake Error at CMakeLists.txt:55 (install): install Library TARGETS given no DESTINATION!. More about this at: https://stackoverflow.com/a/37729574

After 1, CMake will generate (I used Visual Studio 15 2017 Win64 generator) the fasttext.sln file under build dir, which you can open in VS to build but you will get handful of errors:

• To resolve to_string related errors, I added #include <string> in productquantizier.cc
• For LNK1104 can not open file 'pthread.lib', made two more changes in CMakeLists.txt
  1 - Change line 19 from set(CMAKE_CXX_FLAGS " -pthread -std=c++11 -funroll-loops -O3 -march=native") to set(CMAKE_CXX_FLAGS " -std=c++11 -funroll-loops -O3 -march=native")
  2 - Change line 53 from target_link_libraries(fasttext-bin pthread fasttext-static) to target_link_libraries(fasttext-bin fasttext-static)
  I'm not sure how second change will affect the functionality (pthreads (POSIX threads) are POSIX only https://stackoverflow.com/a/2150967)

After all these changes, build will be successful here's my VS build output and CMake log

I did minimal testing with and without thread flag and it worked for both.

With out thread flag:
C:\Users\Saad\Desktop\fastText\build\Debug>fasttext skipgram -input test.txt -output model
Read 0M words
Number of words: 646
Number of labels: 0
Progress: 100.0% words/sec/thread: 293 lr: 0.000000 loss: 2.888118 ETA: 0h 0m

With thread set:
C:\Users\Saad\Desktop\fastText\build\Debug>fasttext skipgram -input test.txt -output model2 -thread 4
Read 0M words
Number of words: 646
Number of labels: 0
Progress: 100.0% words/sec/thread: 397 lr: 0.000000 loss: 2.883805 ETA: 0h 0m

Hope it helps!

P.S - @cpuhrsch can you comment on the changes and how this will effect the resulting fasttext.exe, thanks!

[matanox]

@suamin thanks a lot for the precise depiction. This easily reproduces and works (do you get your fasttext.exe executable built under Debug/ or elsewhere?).

Yet, the generated executable aborts a lot of the time for me, without any helpful error message.

It seems that instead of printing an error message to the console it only aborts. E.g. if the input file name specified for -input is wrong, it will crash rather than output that the file cannot be found. We do not get that behavior on linux ― on linux we get a specific error message before aborting.

This is problematic as in my experimentation oftenly the same fasttext command succeeds and later fails when repeated, and in those cases the error message is really necessary for troubleshooting.

Maybe something about stdout or console output that needs to be adapted for Windows (this repeatedly happens in both the plain CMD console of Windows 10 and in the so-called "Visual Studio Developer Command Prompt", as well after restarting the OS). Or it is actually some Windows setting I'm not aware of, and have never tweaked, that relates to how program stderr is handled by the OS (I seldom use my Windows partition and wouldn't know).

[xiamx]

It might be possible to port the parallelism to OpenMP, then both posix systems and windows can be supported

…

________________________________ From: matanster <notifications@github.com> Sent: Tuesday, January 16, 2018 4:45:36 PM To: facebookresearch/fastText Cc: Mengxuan Xia; Mention Subject: Re: [facebookresearch/fastText] Support building and running on Windows (#39) @suamin<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsuamin&data=02%7C01%7C%7C36f4f8e7ce1e402905ab08d55d2a7bd0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636517359394548102&sdata=6qlqizHirfpLIg6WrjzFHJzndWSl1qdfFpYF6fonric%3D&reserved=0> Do you get your fasttext.exe executable built under Debug/ or elsewhere? I get arbitrary crashes when using -pretrainedVectors with latest pretrained vectors provided by facebook. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffacebookresearch%2FfastText%2Fpull%2F39%23issuecomment-358117588&data=02%7C01%7C%7C36f4f8e7ce1e402905ab08d55d2a7bd0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636517359394548102&sdata=EObklW%2Ft92sEOs%2FoeOM4LWMwQaSiVjNcZOP0dWS8VJc%3D&reserved=0>, or mute the thread<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAnTR2JYvF2vjkt5afsTUKlEFHYv9K0xks5tLRiAgaJpZM4JnhnI&data=02%7C01%7C%7C36f4f8e7ce1e402905ab08d55d2a7bd0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636517359394548102&sdata=PcdUZRUASY%2FDdBil8kNeVt4Zoa7Q2v5PcQN3cHc52Uc%3D&reserved=0>.

[suamin]

@matanster Yes, you can get the fasttext.exe under Debug, MinSizRel, Release and RelWithDebInfo depending on your solution configurations. Also, thanks for reporting the crash issues. I also faced them after building the solution and early testing but didn't had much time to check the crash causes before.

I'm not very sure if this has anything to do with Windows but I can acknowledge std:cerr is working fine. You can also verify it by passing in different args e.g this command fasttext skipgram input data.txt output model prints Provided argument without a dash! Usage: followed by Usage menu as per line 94 in args.cc.

The issue is with exceptions that are being thrown and they are not well handled in the code. First the /EHsc flag should be set in CMAKE_CXX_FLAGS, although when not set it will be issued as warning after build and not as an error but since exceptions are raised there including EH should be done. (Note: its debatable between EHsc and EHa). Second, you will need to add try { /* */ } catch (const std::exception& e) { /* */ } blocks wherever the exception is thrown to resolve the crashing.

Example Case:
when -input is wrong fastText throws an exception, defined at line 655 (fasttext.cc), under train function which is being called from main.cc line 295. If you replace this line by try catch block:

try {
	  fasttext.train(a);
  } catch (const std::invalid_argument& e){
	  std::cerr << e.what() << std::endl;
	  exit(EXIT_FAILURE);
  }

and re-do the whole steps (configure, generate (cmake) and build (VS)) and given an invalid -input then it will not crash and prints to console INVALID_FNAME cannot be opened for training! similarly if any other std::invalid_argument exceptions are thrown by train func they will be caught and message will be printed (another example: fasttext skipgram -input - -output model will print Cannot use stdin for training! to console)

I haven't gone through all the source files but if this is the cause (which apparently definitely looks like) then try catch blocks will be needed or if there is a neat hack to automatically redirect all std::<exceptions> to std:cerr then please share, thanks!

@xiamx yes that makes sense. I think pthreads can be supported if subsytem is set to POSIX (/SUBSYSTEM:POSIX) under project > linker > system settings.

Last note, without pthreads flag I was worried if multiple threads are actually utilized or not with resulting executable, but they are:

In this use case I set the -thread flag to be 20.

[xiamx]

I might be wrong, /SUBSYSTEM:POSIX support was removed in modern version of windows

[xiamx]

Might be helpful for those who don't want to build fastText locally on windows. My fork now has fastText automatically built with CI, tracking master branch https://github.com/xiamx/fastText/releases