automatic installation not working

[Teque5]

Yo nice tool, but I think auto install isn't quite working yet. This is Ubuntu 18.04.

It seems like /var/app/warp-cli/wdt should point to /var/app/warp-cli/build/wdt (setup.py:45) but something else may be wrong. If i try to manually cmake wdt it says folly is not installed. I inspected your setup.py and I didn't see where it was supposed to make folly.

teque5  /  var  app  python3 /var/app/warp-cli/core/warp.py --install /var/app/warp-cli
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.4ubuntu1).
libboost-system-dev is already the newest version (1.65.1.0ubuntu1).
libdouble-conversion-dev is already the newest version (2.0.1-4ubuntu1).
libboost-all-dev is already the newest version (1.65.1.0ubuntu1).
libgoogle-glog-dev is already the newest version (0.3.5-1).
libgtest-dev is already the newest version (1.8.0-6).
libjemalloc-dev is already the newest version (3.6.0-11).
cmake is already the newest version (3.10.2-1ubuntu2.18.04.1).
libssl-dev is already the newest version (1.1.1-1ubuntu2.1~18.04.4).
openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Cloning into 'folly'...
remote: Enumerating objects: 193, done.
remote: Counting objects: 100% (193/193), done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 64891 (delta 81), reused 85 (delta 38), pack-reused 64698
Receiving objects: 100% (64891/64891), 26.07 MiB | 1.45 MiB/s, done.
Resolving deltas: 100% (52400/52400), done.
Cloning into 'wdt'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 4695 (delta 7), reused 16 (delta 5), pack-reused 4671
Receiving objects: 100% (4695/4695), 1.77 MiB | 863.00 KiB/s, done.
Resolving deltas: 100% (3758/3758), done.
CMake Error: The source directory "/var/app/warp-cli/wdt" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
Install Complete!

[JustinTimperio]

Sorry for the slow response. I just pushed a commit fixing this path error.

I'm still having issues getting WDT to build on Ubuntu 18.04 though. I'll look more into this and get back to you

[JustinTimperio]

So I have confirmed that the cmake path error is fixed but unfortunately FOLLY is returning an error to all distros I test. This is most likely due to a recent update to folly. I have submitted a bug to the WDT github here. While I have fixed the issue you submitted I am going to leave this issue open until I can resolve the build process.

The error will read as follows:

Scanning dependencies of target folly4wdt
[ 2%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/hash/Checksum.cpp.o
[ 4%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/Demangle.cpp.o
[ 6%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/Conv.cpp.o
[ 9%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/hash/detail/Crc32CombineDetail.cpp.o
[ 11%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/ScopeGuard.cpp.o
[ 13%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/lang/ColdClass.cpp.o
[ 15%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/hash/detail/Crc32cDetail.cpp.o
[ 18%] Building CXX object CMakeFiles/folly4wdt.dir/var/app/warp-cli/build/folly/folly/hash/detail/ChecksumDetail.cpp.o
[ 20%] Linking CXX shared library libfolly4wdt.so
[ 20%] Built target folly4wdt
Scanning dependencies of target wdt_min
[ 25%] Building CXX object CMakeFiles/wdt_min.dir/util/EncryptionUtils.cpp.o
[ 25%] Building CXX object CMakeFiles/wdt_min.dir/util/WdtSocket.cpp.o
[ 27%] Building CXX object CMakeFiles/wdt_min.dir/util/ClientSocket.cpp.o
[ 29%] Building CXX object CMakeFiles/wdt_min.dir/util/DirectorySourceQueue.cpp.o
[ 31%] Building CXX object CMakeFiles/wdt_min.dir/Reporting.cpp.o
[ 34%] Building CXX object CMakeFiles/wdt_min.dir/Receiver.cpp.o
[ 36%] Building CXX object CMakeFiles/wdt_min.dir/ErrorCodes.cpp.o
[ 38%] Building CXX object CMakeFiles/wdt_min.dir/ReceiverThread.cpp.o
[ 40%] Building CXX object CMakeFiles/wdt_min.dir/util/FileByteSource.cpp.o
[ 43%] Building CXX object CMakeFiles/wdt_min.dir/util/FileCreator.cpp.o
[ 65%] Building CXX object CMakeFiles/wdt_min.dir/WdtOptions.cpp.o
[ 45%] Building CXX object CMakeFiles/wdt_min.dir/util/ThreadsController.cpp.o
[ 47%] Building CXX object CMakeFiles/wdt_min.dir/util/Stats.cpp.o
[ 50%] Building CXX object CMakeFiles/wdt_min.dir/util/SerializationUtil.cpp.o
[ 52%] Building CXX object CMakeFiles/wdt_min.dir/WdtBase.cpp.o
[ 54%] Building CXX object CMakeFiles/wdt_min.dir/Sender.cpp.o
[ 56%] Building CXX object CMakeFiles/wdt_min.dir/util/ThreadTransferHistory.cpp.o
[ 59%] Building CXX object CMakeFiles/wdt_min.dir/util/CommonImpl.cpp.o
[ 61%] Building CXX object CMakeFiles/wdt_min.dir/util/FileWriter.cpp.o
[ 63%] Building CXX object CMakeFiles/wdt_min.dir/util/TransferLogManager.cpp.o
[ 68%] Building CXX object CMakeFiles/wdt_min.dir/WdtThread.cpp.o
[ 70%] Building CXX object CMakeFiles/wdt_min.dir/WdtResourceController.cpp.o
[ 72%] Building CXX object CMakeFiles/wdt_min.dir/util/ServerSocket.cpp.o
[ 75%] Building CXX object CMakeFiles/wdt_min.dir/Throttler.cpp.o
[ 77%] Building CXX object CMakeFiles/wdt_min.dir/WdtTransferRequest.cpp.o
[ 79%] Building CXX object CMakeFiles/wdt_min.dir/SenderThread.cpp.o
[ 81%] Building CXX object CMakeFiles/wdt_min.dir/Protocol.cpp.o
/var/app/warp-cli/build/wdt/util/TransferLogManager.cpp: In member function ‘std::__cxx11::string facebook::wdt::LogParser::getFormattedTimestamp(int64_t)’:
/var/app/warp-cli/build/wdt/util/TransferLogManager.cpp:752:8: warning: ‘__builtin___snprintf_chk’ output may be truncated before the last format character [-Wformat-truncation=]
string LogParser::getFormattedTimestamp(int64_t timestampMicros) {
^~~~~~~~~
In file included from /usr/include/stdio.h:862:0,
from /usr/include/c++/7/cstdio:42,
from /usr/include/c++/7/ext/string_conversions.h:43,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /usr/include/gflags/gflags.h:81,
from /var/app/warp-cli/build/wdt/../wdt/ErrorCodes.h:11,
from /var/app/warp-cli/build/wdt/../wdt/Protocol.h:11,
from /var/app/warp-cli/build/wdt/../wdt/util/TransferLogManager.h:11,
from /var/app/warp-cli/build/wdt/util/TransferLogManager.cpp:9:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:65:44: note: ‘__builtin___snprintf_chk’ output between 25 and 76 bytes into a destination of size 25
__bos (__s), __fmt, __va_arg_pack ());
^
[ 84%] Linking CXX shared library libwdt_min.so
[ 84%] Built target wdt_min
Scanning dependencies of target wdtbin
Scanning dependencies of target wdt
[ 86%] Building CXX object CMakeFiles/wdt.dir/Wdt.cpp.o
[ 88%] Building CXX object CMakeFiles/wdtbin.dir/util/WdtFlags.cpp.o
[ 93%] Building CXX object CMakeFiles/wdtbin.dir/Wdt.cpp.o
[ 90%] Building CXX object CMakeFiles/wdtbin.dir/wdtCmdLine.cpp.o
[ 95%] Building CXX object CMakeFiles/wdt.dir/util/WdtFlags.cpp.o
[ 97%] Linking CXX shared library libwdt.so
[ 97%] Built target wdt
[100%] Linking CXX executable _bin/wdt/wdt
libfolly4wdt.so: undefined reference to `folly::strlcpy(char*, char const*, unsigned long)'
collect2: error: ld returned 1 exit status
CMakeFiles/wdtbin.dir/build.make:154: recipe for target '_bin/wdt/wdt' failed
make[2]: *** [_bin/wdt/wdt] Error 1
CMakeFiles/Makefile2:180: recipe for target 'CMakeFiles/wdtbin.dir/all' failed
make[1]: *** [CMakeFiles/wdtbin.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

[JustinTimperio]

I have resolved this issue in commit 'af4cab1' by forcing folly to backdate to a stable release. I will leave the current error open on WDT git as this seems to be an actual error in the most recent builds of folly.

[Teque5]

Don't want to rain on your parade but I had to apply the following diff to get things working on Ubuntu 18.04. I picked a random earlier version of folly. I think your setup.py currently just checks out the most recent tag, which still exhibits the problem linked above.

diff --git a/core/setup.py b/core/setup.py
index e5a68dd..9337018 100755
--- a/core/setup.py
+++ b/core/setup.py
@@ -38,9 +38,9 @@ def setup_warp(base_dir, remote_install=False):
         sys.exit('Automated package installs for ' + os_name + ' are not supported.')
 
     ## download and build wdt from source
-    mkdir(base_dir + '/build', 'r')
+    mkdir(base_dir + '/build', 'u')
     os.system('cd ' + base_dir + '/build && git clone https://github.com/facebook/folly.git')
-    os.system('cd ' + base_dir + '/build/folly && git checkout "$(git describe --abbrev=0 --always)"')
+    os.system('cd ' + base_dir + '/build/folly && git checkout v2019.09.23.00')
     os.system('cd ' + base_dir + '/build && git clone https://github.com/facebook/wdt.git')
     os.system('mkdir ' + base_dir + '/build/wdt/_build')
     os.system('cd ' + base_dir + '/build/wdt/_build && cmake -DCMAKE_INSTALL_PREFIX="/usr" -DCMAKE_BUILD_TYPE=Release ../ && make -j && sudo make install')

[JustinTimperio]

Thanks again for submitting. It looks like you are right on both counts. I should have unit tested my last release otherwise I would have caught that root error.

The folly checkout error while unfortunate was intentional. I was hoping to avoid manually specifying a build date so it could automatically remain updated. For the time being, I will set this to manually specify folly release v2019.09.30.00.

[JustinTimperio]

I just merged an update from my dev branch which I unit tested on Ubuntu Server 18.04, Ubuntu Desktop 19.04, Fedora 30 Workstation, and Debian 9.5. I will close this issue for now but I expect that folly will continue to provide errors during the build process moving forward.