Following Quickstart failes with docker build issue #556
Comments
|
seems this might be related to https://stackoverflow.com/questions/66319610/gpg-error-in-ubuntu-21-04-after-second-apt-get-update-during-docker-build - I was trying this on a 12 core 64 GB Mac with an older docker version. I will retry on an ubuntu machine. |
|
Hello and thanks for your interest in the QLever project. Your stack overflow link says, that possible fixes are a newer Docker version or an older base image. In the We have never tried using QLever on a machine where the native OS is not a Linux, but we will be interested in how well this works. |
|
Thx for the quick response date;docker build --file Dockerfiles/Dockerfile.Ubuntu20.04 -t qlever .;date
Thu Jan 27 10:21:46 CET 2022
[+] Building 1484.6s (21/25)
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load build definition from Dockerfile.Ubuntu20.04 0.0s
=> => transferring dockerfile: 2.47kB 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:20.04 1.8s
=> [base 1/6] FROM docker.io/library/ubuntu:20.04@sha256:b5a61709a9a4428 6.0s
=> => resolve docker.io/library/ubuntu:20.04@sha256:b5a61709a9a44284d88f 0.0s
=> => sha256:ea362f368469f909a95f9a6e54ebe0121ce0a8e3c 28.57MB / 28.57MB 4.4s
=> => sha256:b5a61709a9a44284d88fb12e5c48db0409cfad5b69d 1.42kB / 1.42kB 0.0s
=> => sha256:57df66b9fc9ce2947e434b4aa02dbe16f6685e20db0c170 529B / 529B 0.0s
=> => sha256:d13c942271d66cb0954c3ba93e143cd253421fe0772 1.46kB / 1.46kB 0.0s
=> => extracting sha256:ea362f368469f909a95f9a6e54ebe0121ce0a8e3c30583dd 1.3s
=> [internal] load build context 0.4s
=> => transferring context: 437.98kB 0.3s
=> [base 2/6] RUN apt-get update 6.4s
=> [base 3/6] RUN apt install -y software-properties-common 39.4s
=> [base 4/6] RUN add-apt-repository -y ppa:mhier/libboost-latest 4.4s
=> [base 5/6] RUN add-apt-repository ppa:ubuntu-toolchain-r/test 4.0s
=> [base 6/6] RUN apt-get update 2.8s
=> [builder 1/9] RUN apt-get install -y build-essential cmake libicu-de 38.1s
=> [runtime 1/6] WORKDIR /app 0.0s
=> [runtime 2/6] RUN apt-get update && apt-get install -y wget python3- 21.7s
=> [runtime 3/6] RUN apt install -y lbzip2 libjemalloc-dev libzstd-dev 4.4s
=> [runtime 4/6] RUN groupadd -r qlever && useradd --no-log-init -r -u 1 0.6s
=> [builder 2/9] RUN apt install -y gcc-11 g++-11 17.0s
=> [builder 3/9] RUN apt install -y libjemalloc-dev ninja-build libzstd- 3.8s
=> [builder 4/9] RUN apt install -y libboost1.74-dev 9.9s
=> [builder 5/9] COPY . /app/ 2.9s
=> [builder 6/9] WORKDIR /app/ 0.0s
=> [builder 7/9] WORKDIR /app/build/ 0.0s
=> [builder 8/9] RUN cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COM 1348.0s
=> => # [81/571] Building CXX object CMakeFiles/PrefixHeuristicEvaluatorMain.d
=> => # ir/src/PrefixHeuristicEvaluatorMain.cpp.o
=> => # [82/571] Building CXX object third_party/abseil-cpp/absl/strings/CMake
=> => # Files/absl_cord.dir/cord.cc.o
=> => # [83/571] Building CXX object third_party/antlr4/runtime/Cpp/runtime/CM
=> => # akeFiles/antlr4_static.dir/src/BufferedTokenStream.cpp.o
Thu Jan 27 10:46:31 CET 2022finished after 1484 secs The environment is a Mid 2010 Mac pro Running Mac OS 10.13.6 High Sierra. The docker environment ist docker desktop community version 2.5.0.1 (49550) with Engine 19.03.13 |
|
Was this a succesful run? I am missing step 9/9 and some kind of success message, but there also is no error and I am not used to the output of this particular Docker version. Just let us know, if you have further questions, or a particular use case for QLever. |
|
@joka921 just tried to send you a personal mail regarding https://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData. Would you like to see a separate ticket for adding your solution to https://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData#Performance_Reports ? |
|
After the hints from #563 I adapted the script from #562 for Mac OS with an old docker. The result seems a bit strange: ./qlever -aw
operating system
ProductName: Mac OS X
ProductVersion: 10.13.6
BuildVersion: 17G14033
docker version
Docker version 19.03.13, build 4484c46d9d
memory
./qlever: line 102: free: command not found
diskspace
pulling qlever docker image started at Sun Jan 30 14:29:57 CET 2022
Using default tag: latest
latest: Pulling from adfreiburg/qlever
Digest: sha256:036bde7c6e675c3cf34cfa3f87513af8a43dbd83be736ca83f21c244dfee91f2
Status: Image is up to date for adfreiburg/qlever:latest
docker.io/adfreiburg/qlever:latest
pulling qlever docker image finished at Sun Jan 30 14:29:59 CET 2022 after 2 seconds
wikidata dump already downloaded
creating wikidata index started at Sun Jan 30 14:29:59 CET 2022
2022-01-30 13:30:03.292 - INFO: QLever IndexBuilder, compiled on Jan 29 2022 20:19:12
2022-01-30 13:30:03.417 - INFO: You specified the input format: TTL
2022-01-30 13:30:03.442 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2022-01-30 13:30:03.444 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files (see qlever/docs on GitHub)
2022-01-30 13:30:03.444 - INFO: You specified "num-triples-per-partial-vocab = 10,000,000", choose a lower value if the index builder runs out of memory
2022-01-30 13:30:03.446 - INFO: Processing input triples from /dev/stdin ...
2022-01-30 15:28:29.931 - INFO: Input triples processed: 100,000,000
2022-01-30 15:35:05.262 - INFO: Input triples processed: 200,000,000
2022-01-30 15:41:14.692 - INFO: Input triples processed: 300,000,000
2022-01-30 15:47:24.119 - INFO: Input triples processed: 400,000,000
2022-01-30 15:53:49.298 - INFO: Input triples processed: 500,000,000
2022-01-30 16:00:24.711 - INFO: Input triples processed: 600,000,000
2022-01-30 16:06:09.271 - INFO: Input triples processed: 700,000,000
2022-01-30 16:12:37.717 - INFO: Input triples processed: 800,000,000it takes a full 2 h before the first output appears in the log. It's just by chance that i let the computer run after no output showed up after 15 mins ... the docker stats show some 17 GB memory usage and 300 to 800% CPU usage so i assume multiple cores are at work. |
|
There are two phenomena here:
You don't have to use docker. The two Dockerfiles in https://github.com/ad-freiburg/qlever/tree/master/Dockerfiles tell you exactly what you have to install and how (for Ubuntu 18 and for Ubuntu 20), so that you can run QLever natively. |
This is not "log output not showing", QLever indeed takes two hours before it actually starts doing something.
|
|
Thanks, @joka921, for chiming in. The very long wait until something happens never happened to me so far. I am curious what that will turn out to be. (I did experience some wait, but never very long.) Concerning budget: You get an AMD Ryzen 9 5900X machine for around 1.000 €, so not that expensive. And then a few more €€€ for disk space, depending on how much you want. We have machines with 1,7 TB SSD (cheap) and with a 33 TB HD raid (1.500 € extra). |
|
The budget discussion needs to be a postponed for a few day but i am at it. |
|
A lot has happened with the Dockerfile and binary builds in the meantime. If you still have problems building an index, please open a new issue or reopen this one. |
The text was updated successfully, but these errors were encountered: