Consistent, cross-platform binary builds #524
Comments
|
Thanks, @jeremiahpslewis, can you briefly explain (preferably as a sequence of instructions) what exactly it is that we have to do from our side? |
|
@hannahbast I think I can get the PR 95% of the way together with the Yggdrasil community, then I would tag you on the PR so you could give feedback on the build script in case there are optimizations? Going forward, as the project matures and you (potentially?) adopt semantic versioning, your team or I could then keep the Yggdrasil build up to date with your releases, it's usually just updating the version and git commit hash. |
|
@jeremiahpslewis Thanks! Doesn't our Dockerfile provide all the information you need? It lists all the packages that need to be installed and the exact command with which to build the code. All the build options can be found in the CMakeLists.txt, but you can just use that file as is with cmake. Or doesn't the build work that way with Yggdrasil? |
|
@hannahbast Yep, it should, I don't anticipate any issues, unless there are minor tweaks required to make it possible to build on other platforms. :) |
|
@hannahbast Ok, so here is a question...have you ever seen an error like this where everything builds and then the tests fail? |
|
@jeremiahpslewis I have never seen this error before. Johannes (@joka921), can you comment? The actual error message seems to be in the last two lines |
|
@jeremiahpslewis |
|
@jeremiahpslewis , Alternatively: Is there a documentation where I can run the exact toolchain this script uses on our machines, so we can have a faster cycle? |
|
Ok,
|
|
@joka921 There's a great command line interface which drops you into a debug shell for running builds locally, here are the steps (assumes you have docker installed):
|
|
@joka921 I'm working on updating the PR for the aarch64 etc. builds, you are right that I need to move the binaries to the right location |
|
@joka921 Here are the updated Mac build logs: Link: https://gist.github.com/jeremiahpslewis/ce7191f5283156d87a9993b00caed6b2 Highlights: 2021-12-15T17:22:39.3591876Z �[0m�[1m[17:22:39] �[22m
2021-12-15T17:22:39.3593528Z �[0m�[1m[17:22:39] �[22m�[31mHEAD is now at 944ef05 Merge pull request #55 from Flamefire/disable_tests_for_subproject�[39m
2021-12-15T17:22:39.3980342Z �[0m�[1m[17:22:39] �[22m�[31mSubmodule 'extern/gtest' (https://github.com/google/googletest) registered for path 'extern/gtest'�[39m
2021-12-15T17:22:39.4028849Z �[0m�[1m[17:22:39] �[22m�[31mCloning into '/workspace/srcdir/qlever/build/runtime/thirdparty/utfcpp/extern/gtest'...�[39m
2021-12-15T17:22:40.1329209Z �[0m�[1m[17:22:40] �[22m�[31mIn file included from /workspace/srcdir/qlever/test/TupleHelpersTest.cpp:10:�[39m
2021-12-15T17:22:40.1330689Z �[0m�[1m[17:22:40] �[22m�[31m/workspace/srcdir/qlever/test/../src/util/TupleHelpers.h:54:10: error: no viable constructor or deduction guide for deduction of template arguments of 'tuple'�[39m
2021-12-15T17:22:40.1331855Z �[0m�[1m[17:22:40] �[22m�[31m return std::tuple(�[39m
2021-12-15T17:22:40.1332510Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1333761Z �[0m�[1m[17:22:40] �[22m�[31m/workspace/srcdir/qlever/test/TupleHelpersTest.cpp:69:14: note: in instantiation of function template specialization 'ad_tuple_helpers::toUniquePtrTuple<int, std::basic_string<char>, bool>' requested here�[39m
2021-12-15T17:22:40.1335003Z �[0m�[1m[17:22:40] �[22m�[31m auto x = toUniquePtrTuple(3, "kartoffel"s, true);�[39m
2021-12-15T17:22:40.1335661Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1337305Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:678:7: note: candidate template ignored: requirement '_CheckArgsConstructor<true, void>::__enable_implicit()' was not satisfied [with _Tp = <std::unique_ptr<bool>>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Dummy = true]�[39m
2021-12-15T17:22:40.1339017Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, const _Tp& ... __t)�[39m
2021-12-15T17:22:40.1339741Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1341430Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:698:7: note: candidate template ignored: requirement '_CheckArgsConstructor<true, void>::__enable_explicit()' was not satisfied [with _Tp = <std::unique_ptr<bool>>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Dummy = true]�[39m
2021-12-15T17:22:40.1342848Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, const _Tp& ... __t)�[39m
2021-12-15T17:22:40.1343482Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1345256Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:784:9: note: candidate template ignored: substitution failure [with _Tp = <>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Up = <std::unique_ptr<bool>>]: cannot reference member of primary template because deduced class template specialization 'tuple<>' is an explicit specialization�[39m
2021-12-15T17:22:40.1347375Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, _Up&&... __u)�[39m
2021-12-15T17:22:40.1347937Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1349438Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:804:9: note: candidate template ignored: substitution failure [with _Tp = <>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Up = <std::unique_ptr<bool>>]: cannot reference member of primary template because deduced class template specialization 'tuple<>' is an explicit specialization�[39m
2021-12-15T17:22:40.1350916Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, _Up&&... __u)�[39m
2021-12-15T17:22:40.1351489Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1353387Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:851:9: note: candidate template ignored: substitution failure [with _Tp = <>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Tuple = std::unique_ptr<bool>]: cannot reference member of primary template because deduced class template specialization 'tuple<>' is an explicit specialization�[39m
2021-12-15T17:22:40.1355068Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, _Tuple&& __t)�[39m
2021-12-15T17:22:40.1355778Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1357611Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:865:9: note: candidate template ignored: substitution failure [with _Tp = <>, _Alloc = std::unique_ptr<std::basic_string<char>>, _Tuple = std::unique_ptr<bool>]: cannot reference member of primary template because deduced class template specialization 'tuple<>' is an explicit specialization�[39m
2021-12-15T17:22:40.1359351Z �[0m�[1m[17:22:40] �[22m�[31m tuple(allocator_arg_t, const _Alloc& __a, _Tuple&& __t)�[39m
2021-12-15T17:22:40.1359995Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1361082Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:933:1: note: candidate template ignored: could not match 'tuple' against 'unique_ptr'�[39m
2021-12-15T17:22:40.1362320Z �[0m�[1m[17:22:40] �[22m�[31mtuple(allocator_arg_t, const _Alloc&, tuple<_Args...> const&) -> tuple<_Args...>;�[39m
2021-12-15T17:22:40.1363021Z �[0m�[1m[17:22:40] �[22m�[31m^�[39m
2021-12-15T17:22:40.1364164Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:935:1: note: candidate template ignored: could not match 'tuple' against 'unique_ptr'�[39m
2021-12-15T17:22:40.1365319Z �[0m�[1m[17:22:40] �[22m�[31mtuple(allocator_arg_t, const _Alloc&, tuple<_Args...>&&) -> tuple<_Args...>;�[39m
2021-12-15T17:22:40.1365955Z �[0m�[1m[17:22:40] �[22m�[31m^�[39m
2021-12-15T17:22:40.1367136Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:626:5: note: candidate function template not viable: requires 2 arguments, but 3 were provided�[39m
2021-12-15T17:22:40.1368215Z �[0m�[1m[17:22:40] �[22m�[31m tuple(_AllocArgT, _Alloc const& __a)�[39m
2021-12-15T17:22:40.1368869Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1370053Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:616:5: note: candidate function template not viable: requires 1 argument, but 3 were provided�[39m
2021-12-15T17:22:40.1371125Z �[0m�[1m[17:22:40] �[22m�[31m tuple(tuple const&) = default;�[39m
2021-12-15T17:22:40.1371755Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1372894Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:617:5: note: candidate function template not viable: requires 1 argument, but 3 were provided�[39m
2021-12-15T17:22:40.1373801Z �[0m�[1m[17:22:40] �[22m�[31m tuple(tuple&&) = default;�[39m
2021-12-15T17:22:40.1374377Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1375982Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:642:5: note: candidate template ignored: requirement '_CheckArgsConstructor<true, void>::__enable_implicit()' was not satisfied [with _Tp = <std::unique_ptr<int>, std::unique_ptr<std::basic_string<char>>, std::unique_ptr<bool>>, _Dummy = true]�[39m
2021-12-15T17:22:40.1377946Z �[0m�[1m[17:22:40] �[22m�[31m tuple(const _Tp& ... __t) _NOEXCEPT_((__all<is_nothrow_copy_constructible<_Tp>::value...>::value))�[39m
2021-12-15T17:22:40.1378734Z �[0m�[1m[17:22:40] �[22m�[31m ^�[39m
2021-12-15T17:22:40.1380312Z �[0m�[1m[17:22:40] �[22m�[31m/opt/aarch64-apple-darwin20/aarch64-apple-darwin20/sys-root/usr/include/c++/v1/tuple:660:14: note: candidate template ignored: requirement '_CheckArgsConstructor<true, void>::__enable_explicit()' was not satisfied [with _Tp = <std::unique_ptr<int>, std::unique_ptr<std::basic_string<char>>, std::unique_ptr<bool>>, _Dummy = true]�[39m |
|
Latest Mac error: |
|
Thank you for trying this out, it is really interesting to see, what is missing to make QLever work on different platforms. After looking at those logs for quite some time, I can now say the following:
For your concrete issues this means the following:
Could you please comment on the possible target group of the builds? It seems to be people who have no problem using the package manager of a rather uncommon program language, but for some reason cannot use a provided Dockerfile, or Docker image. I currently do not see, that this is a large target group. Best regards |
|
@jeremiahpslewis @joka921 Should we still keep this issue open? |
|
Yes please! We're getting so close to having this done. (I hope.) If you have an Apple Silicon Mac and care to install Julia, you can try out the following: using Pkg
Pkg.add("https://github.com/jeremiahpslewis/qlever_jll.jl.git")
using QLever_jll
using XZ_jll
using Downloads
using JSON
olympics_data_url = "https://github.com/ad-freiburg/qlever/blob/64eb86f9fa30404df56fdae50475d0eb505cbc26/examples/olympics.nt.xz?raw=true"
mktempdir() do tdir
olympics_data_path = joinpath(tdir, "olympics.nt.xz")
Downloads.download(olympics_data_url, olympics_data_path)
settings_path = joinpath(tdir, "settings.json")
open(settings_path,"w") do f
JSON.print(f, Dict("num-triples-per-partial-vocab" => 50000000, "ascii-prefixes-only" => true))
end
run(pipeline(`$(xz()) -d -c $olympics_data_path`, `$(IndexBuilderMain()) -F ttl -f - -l -i olympics -s $settings_path`))
run(`$(ServerMain()) -i olympics -p 7001`) # with debug
# curl -Gs http://localhost:7001 --data-urlencode "query=PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> PREFIX medal: <http://wallscope.co.uk/resource/olympics/medal/> PREFIX olympics: <http://wallscope.co.uk/ontology/olympics/> SELECT ?athlete (COUNT(?medal) as ?count_medal) WHERE { ?medal olympics:medal medal:Gold . ?medal olympics:athlete/rdfs:label ?athlete . } GROUP BY ?athlete ORDER BY DESC(?count_medal) LIMIT 10"
endThis is all that would (will) be necessary to install, build an index, and then spin up a qlever server. Once the remaining issues are ironed out, the same script will work across Mac / linux / windows, no docker required. A further step is then to wrap the qlever_jll binary in a Qlever.jl package (which I would gladly do) and then you'd have something like (very rough, non-idiomatic sketch): using QLever
d = QLever.load_dataset("olympics")
QLever.build_index(d)
Qlever.run_server(d)
The hope is that this meaningfully changes the degree to which Qlever becomes accessible to a broad group of practitioners. :) |
|
@jeremiahpslewis @joka921 Awesome, thanks! By the way, you can now start up a generic QLever UI with this maximally simple command line. You then have the QLever UI running on port 7000 (you can pick any port you want) sending requests to an arbitrary QLever backend on port 7001. I will work more on making this configurable such that you can specify a different backend port via the command line or a configuration for a few standard datasets (like one that contains all the prefixes for Wikidata). |
|
@hannahbast Cool! Unfortunately docker on newer Macs is impossibly slow so that's a hurdle specific to my setup which I'm trying to avoid. But the other idea I'd (personally) like to push is that the 'glue code' which is necessary to orchestrate qlever (download / clean data, trigger index job, perhaps cache an index to the cloud, run queries and load / manipulate results) can be consolidated into a single package. To be fair, there's no reason that Julia has to be the (only) language for doing this, but the binary build/ship support is pretty darn good. |
|
@jeremiahpslewis The QLever UI isn't really doing anything except serving some pages. The UI code is JavaScript and runs on the client. Can you try whether it works on your MAC? All you need is a running QLever instance and docker installed. For the glue code, we currently have a |
|
|
Speaking of pipelines, Which error do you get with the QLever UI command? |
|
Just reran it and it works. No objections to |
|
@jeremiahpslewis I encountered the "sparqlExpression" "referenced in section"/"defined in discarded section" errors when building qlever with manually built GCC 11 and GCC 12 compilers that I built and ran on Ubuntu 20.04.4 LTS. The system was a cluster and I lacked sudo/root to install the GCC 11 or GCC 12 packages, so I had to build them myself. I built the GCC compilers with the Intel C++ Compiler which is based on clang/LLVM.
I have not figured out what was causing those errors. I was able to build the same qlever code using Ubuntu 20.04.4 LTS with the GCC 11 as described in https://github.com/ad-freiburg/qlever/runs/7525347081?check_suite_focus=true using: I have still not found a way to build on systems that lack pre-packaged GCC 11 or GCC 12. I assumed it was due to me doing something wrong when building those compilers. @jeremiahpslewis @joka921, - how did you get past the "sparqlExpression" "referenced in section"/"defined in discarded section" errors? Did you identify a root cause? |
|
@donpellegrino We did not actually fix it, but we switched to clang for the respective Platforms which does not have this problem. I have in the meantime identified how this error is triggered. In happens when compiling these coroutines with G++ with the I have not yet found the time to make a minimal example of this to figure out if this is a bug in G++ which has not yet been reported. |
It would be great to be able to run qlever as a built binary on a range of platforms and without relying upon docker. The PR below will (hopefully) make this possible, using Julia's packaging system.
The text was updated successfully, but these errors were encountered: