Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running tests #2585

Open
alexsaber opened this Issue Apr 5, 2019 · 10 comments

Comments

Projects
None yet
4 participants
@alexsaber
Copy link

alexsaber commented Apr 5, 2019

Question

How to execute all the tests? I want to run exactly the same tests when the CI build server runs.
Can you provide a step-by-step guide for someone without experience in C/C++ development?

What I tried

Like it says in TESTING.MD and COMPILE.MD I tried to execute "make run_all" and "make run_nokdbtests"

The problem

The tests are executed, but a lot more of them fail compared to the output of the build server.

The output

I built the project like this
mkdir build
cd build
cmake -DBUILD_SHARED=OFF -DBUILD_FULL=ON .. --> otherwise I got errors
make -j 5

And I ran tests like this
make run_all

-----a lot of lines omitted here------

-----below is the last seen output-------
The command kdb-full export failed while accessing the key database with the info:
Sorry, the error (#49) occurred ;(
Description: unknown command detected in dumpfile
Reason: key
Ingroup: plugin
Module: dump
At: /home/osboxes/TU/libelektra/src/plugins/dump/dump.cpp:191
Mountpoint: user
Configfile: /home/osboxes/.config/default.ecf

fatal: Could not export user config

64% tests passed, 63 tests failed out of 177

Label Time Summary:
kdbtests = 37.02 secproc (78 tests)
memleak = 36.32 sec
proc (70 tests)

Total Test time (real) = 39.77 sec

The following tests FAILED:
9 - testtool_mergingkdb (Failed)
26 - testcpp_kdb (Failed)
33 - testshell_markdown_base64 (Failed)
35 - testshell_markdown_boolean (Failed)
37 - testshell_markdown_camel (Failed)
40 - testshell_markdown_conditionals (Failed)
41 - testshell_markdown_constants (Failed)
43 - testshell_markdown_csvstorage (Failed)
45 - testshell_markdown_dini (Failed)
47 - testshell_markdown_directoryvalue (Failed)
50 - testshell_markdown_enum (Failed)
57 - testshell_markdown_hexnumber (Failed)
59 - testshell_markdown_hosts (Failed)
61 - testshell_markdown_iconv (Failed)
62 - testmod_ini (Failed)
63 - testshell_markdown_ini (Failed)
65 - testshell_markdown_ipaddr (Failed)
67 - testshell_markdown_line (Failed)
71 - testshell_markdown_mathcheck (Failed)
74 - testshell_markdown_mini (Failed)
76 - testshell_markdown_mmapstorage (Failed)
77 - testshell_markdown_multifile (Failed)
78 - testshell_markdown_network (Failed)
81 - testshell_markdown_ni (Failed)
82 - testshell_markdown_path (Failed)
83 - testshell_markdown_quickdump (Failed)
86 - testshell_markdown_range (Failed)
88 - testshell_markdown_reference (Failed)
90 - testshell_markdown_rename (Failed)
92 - testshell_markdown_shell (Failed)
95 - testshell_markdown_type (Failed)
97 - testshell_markdown_uname (Failed)
101 - testscr_check_distribution (Failed)
103 - testscr_check_error (Failed)
108 - testscr_check_kdb_internal_check (Failed)
110 - testscr_check_merge (Failed)
112 - testscr_check_mount (Failed)
117 - testscr_check_real_world (Failed)
118 - testscr_check_resolver (Failed)
119 - testscr_check_spec (Failed)
122 - testshell_db_changes (Failed)
123 - testshell_host (Failed)
124 - testshell_listtest (Failed)
125 - testshell_mathcheck (Failed)
126 - testshell_profiletest (Failed)
127 - testshell_script (Failed)
128 - testshell_selftest (Failed)
129 - testshell_markdown_kdb_complete (Failed)
130 - testshell_markdown_kdb_global_umount (Failed)
131 - testshell_markdown_kdb_ls (Failed)
132 - testshell_markdown_kdb_find (Failed)
133 - testshell_markdown_tutorial_arrays (Failed)
134 - testshell_markdown_tutorial_cascading (Failed)
135 - testshell_markdown_tutorial_validation (Failed)
136 - testshell_markdown_msr_syntax (Failed)
137 - testshell_markdown_readme_msr (Failed)
138 - testshell_markdown_issue_template (Failed)
139 - testshell_markdown_ini_crash_test (Failed)
173 - testkdb_conflict (Failed)
174 - testkdb_error (Failed)
175 - testkdb_nested (Failed)
176 - testkdb_simple (Failed)
177 - testkdb_highlevel (Failed)
Errors while running CTest
tests/CMakeFiles/run_all.dir/build.make:57: recipe for target 'tests/CMakeFiles/run_all' failed
make[3]: *** [tests/CMakeFiles/run_all] Error 8
CMakeFiles/Makefile2:12208: recipe for target 'tests/CMakeFiles/run_all.dir/all' failed
make[2]: *** [tests/CMakeFiles/run_all.dir/all] Error 2
CMakeFiles/Makefile2:12215: recipe for target 'tests/CMakeFiles/run_all.dir/rule' failed
make[1]: *** [tests/CMakeFiles/run_all.dir/rule] Error 2
Makefile:3304: recipe for target 'run_all' failed
make: *** [run_all] Error 2

@alexsaber alexsaber changed the title Running tests Running tests #question Apr 5, 2019

@alexsaber alexsaber changed the title Running tests #question Running tests Apr 5, 2019

@kodebach kodebach added the question label Apr 6, 2019

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Apr 6, 2019

The command kdb-full export failed while accessing the key database with the info:
Sorry, the error (#49) occurred ;(
Description: unknown command detected in dumpfile
Reason: key
Ingroup: plugin
Module: dump
At: /home/osboxes/TU/libelektra/src/plugins/dump/dump.cpp:191
Mountpoint: user
Configfile: /home/osboxes/.config/default.ecf

This error message looks like the file /home/osboxes/.config/default.ecf is broken. If you don't have anything important in your KDB, just delete the file, otherwise please post the file so I can take a closer look. (Note: the file is binary, so opening it in a text editor and copying the contents might not work)

It might also be a permissions problem. For development purposes, I would recommend using the cmake options: -DKDB_DB_SYTEM="~/.config/kdb/system" and -DKDB_DB_SPEC="~/.config/kdb/spec".


The easiest way to execute all tests similar to the way that the Jenkins Server does is by following these instructions to build/download and run the debian-stretch docker container (the docker run command has to be executed in the root directory of the git repository). Then inside the container you just run:

mkdir build-docker && cd build-docker

# compile
cmake .. -DBINDINGS="ALL;-DEPRECATED" -DPLUGINS="ALL;-DEPRECATED" -DTOOLS="ALL" -DENABLE_DEBUG=ON -DKDB_DB_HOME="$PWD" -DKDB_DB_SYSTEM="$PWD/.config/kdb/system" -DKDB_DB_SPEC="$PWD/.config/kdb/system"
make -j 10

# run all tests
make run_all

# run tests with valgrind
make run_memcheck

The build server also runs the tests for the installed version of kdb and some other things that are more complicated to replicate. If you really want to run everything (almost never necessary) take a look at the Jenkinsfile.

@alexsaber

This comment has been minimized.

Copy link
Author

alexsaber commented Apr 6, 2019

@kodebach
Thank you very much for such an extended answer!
I followed your advice and went for testing using Docker. I could successfully build my own image by execution following
docker build -t buildelektra-stretch-full \ --build-arg JENKINS_USERID=id -u \ --build-arg JENKINS_GROUPID=id -g \ -f scripts/docker/debian/stretch/Dockerfile \ scripts/docker/debian/stretch/

Then I started a container like this
docker run -it --rm \ -v "$PWD:/home/jenkins/workspace" \ -w /home/jenkins/workspace \ buildelektra-stretch-full

Inside the container, I built the project as you described it, but I get the following error when it is almost finished:

[ 97%] Generating libHShaskell-ghc8.0.1.so
cabal: Error: some packages failed to install:
libelektra-haskell-0.8.26 failed during the final install step. The exception
was:
ExitFailure 1
unrecognized 'configure' option `--dynlibdir=$libdir/$abi'
src/plugins/haskell/CMakeFiles/haskell.dir/build.make:64: recipe for target 'src/plugins/haskell/libHShaskell-ghc8.0.1.so' failed
make[2]: *** [src/plugins/haskell/libHShaskell-ghc8.0.1.so] Error 1
CMakeFiles/Makefile2:5313: recipe for target 'src/plugins/haskell/CMakeFiles/haskell.dir/all' failed
make[1]: *** [src/plugins/haskell/CMakeFiles/haskell.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

Do you have an idea what might cause the error?

@sanssecours

This comment has been minimized.

Copy link
Member

sanssecours commented Apr 7, 2019

Inside the container, I built the project as you described it, but I get the following error when it is almost finished…

I also used the steps described above and the build system reports the same error.

Do you have an idea what might cause the error?

The problem seems to be the Haskell binding. I would recommend to not include this binding:

cmake ..                                        \
      -DBINDINGS="ALL;-DEPRECATED;-haskell"     \
      -DPLUGINS="ALL;-DEPRECATED"               \
      -DTOOLS="ALL"                             \
      -DENABLE_DEBUG=ON                         \
      -DKDB_DB_HOME="$PWD"                      \
      -DKDB_DB_SYSTEM="$PWD/.config/kdb/system" \
      -DKDB_DB_SPEC="$PWD/.config/kdb/system"

. If I use the commands above, the build step (make -j 10) finishes successfully.

@markus2330

This comment has been minimized.

Copy link
Contributor

markus2330 commented Apr 7, 2019

@sanssecours thank you for the info. Can you please fix the Dockerfiles in our repo? (exclude haskell)

@sanssecours

This comment has been minimized.

Copy link
Member

sanssecours commented Apr 7, 2019

Can you please fix the Dockerfiles in our repo? (exclude haskell)

Since the Haskell build still works on the Jenkins build server, I do not think that is a good idea.

@markus2330

This comment has been minimized.

Copy link
Contributor

markus2330 commented Apr 7, 2019

I think it is more important that building Elektra works for new developer than continuously testing something which is not maintained anyway (which is unfortunately the case for Haskell).

If it is again maintained, we readd it to the tests of course. Then such problems can also be fixed by the Haskell maintainer.

@sanssecours

This comment has been minimized.

Copy link
Member

sanssecours commented Apr 7, 2019

I think it is more important that building Elektra works for new developer than continuously testing something which is not maintained anyway (which is unfortunately the case for Haskell).

While I agree, I would not recommend the full Dockerfile for Debian for new developers anyway. In my opinion it makes much more sense to use the Alpine Docker image, which is

  • much smaller,
  • takes a lot less time to build,
  • uses less outdated packages

.

@kodebach

This comment has been minimized.

Copy link
Contributor

kodebach commented Apr 7, 2019

I would not recommend the full Dockerfile for Debian for new developers anyway

Me neither, but the question was how to execute all tests. AFAIK that is only possible with the Debian image.

@alexsaber

This comment has been minimized.

Copy link
Author

alexsaber commented Apr 7, 2019

Thank you very much! I can now execute all tests.

@markus2330

This comment has been minimized.

Copy link
Contributor

markus2330 commented Apr 7, 2019

@sanssecours you can also add that the official repo has Elektra packages as advantage. Seems like the image is also used already, see #2593

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.