Compilation error: /usr/bin/ld: cannot find -lz, -lstdc++

[oushujun]

Hello @maickrau,

I am trying to compile MBG from source. I followed the readme and encounter two errors:

$ make bin/MBG 
g++ -o bin/MBG obj/MBG.o obj/fastqloader.o obj/CommonUtils.o obj/main.o obj/MBGCommon.o obj/FastHasher.o obj/SparseEdgeContainer.o obj/HashList.o obj/UnitigGraph.o obj/BluntGraph.o obj/HPCConsensus.o obj/ErrorMaskHelper.o obj/CompressedSequence.o obj/ConsensusMaker.o obj/StringIndex.o obj/RankBitvector.o obj/UnitigResolver.o obj/UnitigHelper.o obj/BigVectorSet.o -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue -Wl,-Bstatic -lz -Wl,-Bdynamic -Wl,--as-needed -lpthread -pthread -static-libstdc++
/usr/bin/ld: cannot find -lz
/usr/bin/ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
make: *** [makefile:24: bin/MBG] Error 1

Am I missing any prerequisites? Thanks!

Shujun

[ptrebert]

It looks like you are missing zlib and libstdc. It depends on your system how to install those, e.g., on a current Ubuntu (20.04), you could try

sudo apt-get install build-essential zlib1g-dev

[oushujun]

I am a non-root user of our HPC. Can I install them via conda?
I tried the following commands but conda suggested they are already installed in the env.

conda install -c anaconda libstdcxx-ng     
conda install -c anaconda zlib

I tried make clean and recompiled with the conda env that has zlib and libstdc installed, and reported the same error.

Thank you.
Shujun

[ptrebert]

ah, good to know that you are working in a Conda env... so the linker is not finding these libraries assuming they are properly installed in the environment. With the environment being active, can you run the following...

ld --verbose | grep SEARCH_DIR
echo $LD_LIBRARY_PATH
echo $LIBRARY_PATH

and add the output here.

[oushujun]

Thanks for the instant response. Following are the outputs:

(asm) [sou6@login02 MBG]$ ld --verbose | grep SEARCH_DIR
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/x86_64-redhat-linux/lib"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib");
(asm) [sou6@login02 MBG]$ echo $LD_LIBRARY_PATH
/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.3.0/openmpi-3.1.6-rk3nyoehbq3pke4zy4hn7unns3ujtutx/lib:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib
(asm) [sou6@login02 MBG]$ echo $LIBRARY_PATH
/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.3.0/openmpi-3.1.6-rk3nyoehbq3pke4zy4hn7unns3ujtutx/lib:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib

[ptrebert]

Hm, looks like there is a lot going on in your env that should enable you to compile new tools, but since I don't know the system you are working on, let's try to stick to the Conda environment; the Conda environment does not seem to show up in these lists, so try the following:

make clean  # in case you haven't
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=$LD_LIBRARY_PATH
make bin/MBG

I also created a Conda env with these two packages (libstdcxx-ng and zlib), and the library files are located where expected:

ls -1 $CONDA_PREFIX/lib
[...]
libstdc++.so
libstdc++.so.6
libstdc++.so.6.0.29
[...]
libz.a
libz.so
libz.so.1
libz.so.1.2.11
[...]

which should also be the case for your Conda env.

[oushujun]

It works! I also received a bunch of warnings:

/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/envs/asm/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001

Do I need to do anything about them?

Thanks,
Shujun

[ptrebert]

A rough google search suggests to me that this could be a result of your build env not being entirely encapsulated. Are you using a system compiler, or does your environment include one? To make things faster, I tested the source build with the following minimal environment:

name: mbg
channels:
  - conda-forge
dependencies:
  - libstdcxx-ng
  - zlib
  - gcc

Set LD_LIBRARY_PATH=$CONDA_PREFIX/lib // LIBRARY_PATH=$CONDA_PREFIX/lib (= exclude your default HPC env), run make bin/MBG and - optionally - copy the MBG binary to $CONDA_PREFIX/bin to have it available when activating the Conda environment. This worked on my system.

[oushujun]

Here's what I tried:

# setup environment
conda create -n mbg -y
conda activate mbg
conda install libstdcxx-ng zlib gcc git -c conda-forge -y

# check compiler, looks like a number of gcc paths from the HPC.
$ LD_LIBRARY_PATH
-bash: /cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.3.0/openmpi-3.1.6-rk3nyoehbq3pke4zy4hn7unns3ujtutx/lib:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib64:/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/lib: No such file or directory

# check conda gcc path and set paths
$ which gcc
~/bin/miniconda3/envs/mbg/bin/gcc
$ $CONDA_PREFIX/lib
-bash: /home/sou6/bin/miniconda3/envs/mbg/lib: Is a directory
$ LD_LIBRARY_PATH=$CONDA_PREFIX/lib
$ LIBRARY_PATH=$CONDA_PREFIX/lib

# compile MBG
$ make clean
$ make bin/MBG
g++ -c -o obj/MBG.o src/MBG.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
src/MBG.cpp: In lambda function:
src/MBG.cpp:1149:12: warning: unused variable ‘readPosExpandedStart’ [-Wunused-variable]
     size_t readPosExpandedStart = poses[readPos];
            ^~~~~~~~~~~~~~~~~~~~
src/MBG.cpp:1150:12: warning: unused variable ‘readPosExpandedEnd’ [-Wunused-variable]
     size_t readPosExpandedEnd = poses[readPos+kmerSize];
            ^~~~~~~~~~~~~~~~~~
...
g++ -o bin/MBG obj/MBG.o obj/fastqloader.o obj/CommonUtils.o obj/main.o obj/MBGCommon.o obj/FastHasher.o obj/SparseEdgeContainer.o obj/HashList.o obj/UnitigGraph.o obj/BluntGraph.o obj/HPCConsensus.o obj/ErrorMaskHelper.o obj/CompressedSequence.o obj/ConsensusMaker.o obj/StringIndex.o obj/RankBitvector.o obj/UnitigResolver.o obj/UnitigHelper.o obj/BigVectorSet.o -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue -Wl,-Bstatic -lz -Wl,-Bdynamic -Wl,--as-needed -lpthread -pthread -static-libstdc++
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/miniconda3/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001

Looks like it is still using the system gcc. Is that an easy way to hard code the gcc path in the makefile?

[ptrebert]

Two remarks:

1. I could simply be wrong about the cause b/c I just very briefly googled for it :)
2. the Makefile uses whatever compiler is available as cc, i.e. in the active conda environment, what's the output of the following command:

readlink -f `which cc`

For my test setup, this gives /tmp/build_env/bin/x86_64-conda-linux-gnu-gcc. In any case, if MBG is working irrespective of these warnings, the problem is somewhat solved.

[oushujun]

Yes, this is how I am moving forward. I am currently using the version compiled with warning messages and see if it works. I will let you know. Thanks!

Shujun

[oushujun]

It does not work and gives new errors. Not sure if it's related to the compilation:

[Thu Feb 17 19:54:27 2022]
Error in rule buildGraph:
    jobid: 4
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err (check log file(s) for error message)
    shell:
        
cd 1-buildGraph

cat > ./buildGraph.sh <<EOF
#!/bin/sh
set -e

#  Figure out what reads we're going to use.  If correction is enabled, the
#  reads are in a single file that is a relative path away from us; but if it
#  is not enabled, the reads are in possibly multiple files that are at an
#  absolute path.
#
iopt=""

echo "Building graph with reads:"

if [ "True" = "True" ] ; then
  echo "  ../0-correction/hifi-corrected.fasta"
  iopt="-i ../0-correction/hifi-corrected.fasta"
else
  for rf in /home/sou6/oushujun/PacBio_readsets/Tanz/Tanz-1.q20.fastq.gz ; do
    echo "  \$rf"
    iopt="\$iopt -i \$rf"
  done
fi

echo ""
echo ""

#  Build the graph.
#    (rule build_graph in the original)
#
/home/sou6/bin/MBG/bin/MBG \\
  \$iopt \\
  -t 8 \\
  -k 1001 \\
  -r 15000 \\
  -w 100 \\
  --kmer-abundance 1 \\
  --unitig-abundance 2 \\
  --error-masking=collapse-msat \\
  --output-sequence-paths ../1-buildGraph/paths.gaf \\
  --out ../1-buildGraph/hifi-resolved.gfa

#  Find coverage.
#    (rule hifi_coverage_csv in the original Snakefile)
#    (hifi-resolved.gfa -> hifi_nodecov.csv)
#    (\$6 != "" is from 9e31a602925a477a7e52c277eda143e7bd20e52b)
#
awk 'BEGIN \\
     { \\
        FS="[ \t]+"; OFS="\t"; \\
        print "node", "length", "coverage"; \\
     } \\
     \$1 == "S" \\
     { \\
        if (\$6 != "") {
          \$4 = \$6;
        }
        print \$2, length(\$3), substr(\$4, 6); \\
     }' \\
< ../1-buildGraph/hifi-resolved.gfa \\
> ../1-buildGraph/hifi_nodecov.csv
EOF

chmod +x ./buildGraph.sh

./buildGraph.sh > ../1-buildGraph/buildGraph.err 2>&1
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message

1-buildGraph/buildGraph.err

MBG: src/UnitigResolver.cpp:2093: bool canTrimRecursive(ResolvableUnitigGraph&, std::vector<PathGroup>&, std::pair<long unsigned int, bool>, size_t): Assertion `overlap < resolvableGraph.unitigs[pos.first].size()' failed.
./buildGraph.sh: line 39: 91923 Aborted                 (core dumped) /home/sou6/bin/MBG/bin/MBG $iopt -t 8 -k 1001 -r 15000 -w 100 --kmer-abundance 1 --unitig-abundance 2 --error-masking=collapse-msat --output-sequence-paths ../1-buildGraph/paths.gaf --out ../1-buildGraph/hifi-resolved.gfa

[oushujun]

Not sure if it's relevant, the HPC is using CentOs 8.

I notice the compilation was using g++ not gcc. So I installed g++ in the conda env:
conda install -c conda-forge gxx

Check the default gcc and g++:
$ readlink -f `which gcc`
/home/sou6/bin/miniconda3/envs/mbg/bin/x86_64-conda-linux-gnu-gcc
$ readlink -f `which g++`
/home/sou6/bin/miniconda3/envs/mbg/bin/x86_64-conda-linux-gnu-g++

Also export their path to $CC and $CXX:

export CC=/home/sou6/bin/miniconda3/envs/mbg/bin/x86_64-conda-linux-gnu-gcc   
export CXX=/home/sou6/bin/miniconda3/envs/mbg/bin/x86_64-conda-linux-gnu-g++

Try to remake but with fatal error:

$ export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
$ export LIBRARY_PATH=$LD_LIBRARY_PATH
$ make bin/MBG
/home/sou6/bin/miniconda3/envs/mbg/bin/x86_64-conda-linux-gnu-g++ -c -o obj/MBG.o src/MBG.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
In file included from src/fastqloader.h:6,
                 from src/MBG.cpp:13:
zstr/src/zstr.hpp:14:10: fatal error: zlib.h: No such file or directory
   14 | #include <zlib.h>
      |          ^~~~~~~~
compilation terminated.
make: *** [makefile:30: obj/MBG.o] Error 1

As you previously suggested, zlib is installed via conda. Any ideas?

[ptrebert]

ok, first of all, it would be helpful to keep distinct problems in distinct github issues to avoid confusion about what is being discussed. With that in mind, reading the log messages you posted strongly suggests that your problem is this assertion failure:

MBG: src/UnitigResolver.cpp:2093: bool canTrimRecursive(ResolvableUnitigGraph&, std::vector<PathGroup>&, std::pair<long unsigned int, bool>, size_t): Assertion `overlap < resolvableGraph.unitigs[pos.first].size()' failed.
./buildGraph.sh: line 39: 91923 Aborted                 (core dumped) /home/sou6/bin/MBG/bin/MBG $iopt -t 8 -k 1001 -r 15000 -w 100 --kmer-abundance 1 --unitig-abundance 2 --error-masking=collapse-msat --output-sequence-paths ../1-buildGraph/paths.gaf --out ../1-buildGraph/hifi-resolved.gfa

That does not look like it would be related to the warnings during the compilation process. However, to hopefully bring closure to this compilation discussion, please follow these steps precisely from scratch:

conda create --yes -p ./build_env -c conda-forge gcc zlib libstdcxx-ng
conda activate ./build_env
readlink -f `which cc`  # note: it's "cc" - this should point to the compiler in your conda env
[ clone / set up and init MBG repo as described in the README - do not call make]
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib  # note: no reference to your system's default LD_LIBRARY_PATH
export LIBRARY_PATH=$LD_LIBRARY_PATH
echo $LD_LIBRARY_PATH  # this should show only one single path, i.e. the lib directory in your conda env
make bin/MBG
cp  bin/MBG $CONDA_PREFIX/bin
MBG --help  # should show MBG's help

Does that result in a build process w/o the above warnings?

[oushujun]

I thought this is the same compilation issue, so I didn't open a new issue. I am sorry if the assertion failure is a different problem. I followed your commands precisely, and below is the STDOUT:

(/home/sou6/bin/MBG/build_env) [sou6@devschatz1 MBG]$ echo $LD_LIBRARY_PATH
/home/sou6/bin/MBG/build_env/lib
(/home/sou6/bin/MBG/build_env) [sou6@devschatz1 MBG]$ make bin/MBG
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/MBG.o src/MBG.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
src/MBG.cpp: In lambda function:
src/MBG.cpp:1149:12: warning: unused variable 'readPosExpandedStart' [-Wunused-variable]
 1149 |     size_t readPosExpandedStart = poses[readPos];
      |            ^~~~~~~~~~~~~~~~~~~~
src/MBG.cpp:1150:12: warning: unused variable 'readPosExpandedEnd' [-Wunused-variable]
 1150 |     size_t readPosExpandedEnd = poses[readPos+kmerSize];
      |            ^~~~~~~~~~~~~~~~~~
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/fastqloader.o src/fastqloader.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/CommonUtils.o src/CommonUtils.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/main.o src/main.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue -DVERSION="\"Branch master commit 03b4de16d48f0fb2c7b42f682057dc6a24fb6967 2022-02-17 16:17:11 -0500\""
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/MBGCommon.o src/MBGCommon.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/FastHasher.o src/FastHasher.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/SparseEdgeContainer.o src/SparseEdgeContainer.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/HashList.o src/HashList.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/UnitigGraph.o src/UnitigGraph.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/BluntGraph.o src/BluntGraph.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/HPCConsensus.o src/HPCConsensus.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
src/HPCConsensus.cpp: In lambda function:
src/HPCConsensus.cpp:22:19: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   22 |    if (compressed >= 0 && compressed <= 3)
      |        ~~~~~~~~~~~^~~~
src/HPCConsensus.cpp:36:19: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   36 |    if (compressed >= 0 && compressed <= 3)
      |        ~~~~~~~~~~~^~~~
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/ErrorMaskHelper.o src/ErrorMaskHelper.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
In file included from /data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/include/c++/9.3.0/cassert:44,
                 from src/TwobitLittleBigVector.h:4,
                 from src/CompressedSequence.h:9,
                 from src/MBGCommon.h:8,
                 from src/ErrorMaskHelper.h:7,
                 from src/ErrorMaskHelper.cpp:5:
src/ErrorMaskHelper.cpp: In function 'std::pair<short unsigned int, long unsigned int> getCodeAndRunlength(const SequenceCharType&, size_t, size_t, uint16_t)':
src/ErrorMaskHelper.cpp:105:17: warning: comparison is always true due to limited range of data type [-Wtype-limits]
  105 |   assert(str[i] >= 0 && str[i] <= 3);
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/CompressedSequence.o src/CompressedSequence.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/ConsensusMaker.o src/ConsensusMaker.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/StringIndex.o src/StringIndex.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
src/StringIndex.cpp: In member function 'std::string StringIndex::getString(uint16_t, uint32_t) const':
src/StringIndex.cpp:30:17: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   30 |  if (compressed >= 0 && compressed <= 3)
      |      ~~~~~~~~~~~^~~~
src/StringIndex.cpp: In member function 'uint32_t StringIndex::getIndex(uint16_t, std::variant<long unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >)':
src/StringIndex.cpp:70:17: warning: comparison is always true due to limited range of data type [-Wtype-limits]
   70 |  if (compressed >= 0 && compressed <= 3)
      |      ~~~~~~~~~~~^~~~
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/RankBitvector.o src/RankBitvector.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/UnitigResolver.o src/UnitigResolver.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
In file included from /data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/include/c++/9.3.0/cassert:44,
                 from zstr/src/strict_fstream.hpp:3,
                 from zstr/src/zstr.hpp:15,
                 from src/fastqloader.h:6,
                 from src/ReadHelper.h:11,
                 from src/UnitigResolver.h:10,
                 from src/UnitigResolver.cpp:7:
src/UnitigResolver.cpp: In function 'void replacePaths(ResolvableUnitigGraph&, std::vector<PathGroup>&, const BigVectorSet&, const phmap::flat_hash_map<std::pair<std::pair<long unsigned int, bool>, std::pair<long unsigned int, bool> >, long unsigned int>&)':
src/UnitigResolver.cpp:1211:27: warning: comparison of unsigned expression >= 0 is always true [-Wtype-limits]
 1211 |   assert(nodePosStarts[0] >= 0);
src/UnitigResolver.cpp: In function 'void resolveRound(ResolvableUnitigGraph&, std::vector<PathGroup>&, const HashList&, size_t, size_t, size_t, size_t)':
src/UnitigResolver.cpp:2320:10: warning: unused variable 'newSize' [-Wunused-variable]
 2320 |   size_t newSize = resolvableGraph.unitigs.size();
      |          ^~~~~~~
src/UnitigResolver.cpp:2282:9: warning: variable 'lastTopSize' set but not used [-Wunused-but-set-variable]
 2282 |  size_t lastTopSize = 0;
      |         ^~~~~~~~~~~
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/UnitigHelper.o src/UnitigHelper.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -c -o obj/BigVectorSet.o src/BigVectorSet.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++ -o bin/MBG obj/MBG.o obj/fastqloader.o obj/CommonUtils.o obj/main.o obj/MBGCommon.o obj/FastHasher.o obj/SparseEdgeContainer.o obj/HashList.o obj/UnitigGraph.o obj/BluntGraph.o obj/HPCConsensus.o obj/ErrorMaskHelper.o obj/CompressedSequence.o obj/ConsensusMaker.o obj/StringIndex.o obj/RankBitvector.o obj/UnitigResolver.o obj/UnitigHelper.o obj/BigVectorSet.o -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue -Wl,-Bstatic -lz -Wl,-Bdynamic -Wl,--as-needed -lpthread -pthread -static-libstdc++
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(adler32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(deflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inflate.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inftrees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(trees.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(zutil.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(crc32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/sou6/bin/MBG/build_env/lib/libz.a(inffast.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001

I notice the compilation was not calling the conda gcc but the system g++.

[ptrebert]

What about this line - what is the output of this call (inside the active conda env):

conda activate ./build_env
readlink -f `which cc`  # note: it's "cc" - this should point to the compiler in your conda env

So, I guess then it's reasonable to assume that the mismatch between (system) compiler and the libs in the Conda env lead to these warnings. To confirm, we can fix the Conda compiler in the makefile...

[oushujun]

It points to the conda env:

(/home/sou6/bin/MBG/build_env) [sou6@devschatz1 MBG]$ conda activate ./build_env
(/home/sou6/bin/MBG/build_env) [sou6@devschatz1 MBG]$ readlink -f `which cc`
/home/sou6/bin/MBG/build_env/bin/x86_64-conda-linux-gnu-gcc

The first line of the makefile is GPP=$(CXX), and that points to:

$ echo $CXX
/data/apps/linux-centos8-cascadelake/gcc-9.2.0/gcc-9.3.0-bnvby67rgbqevwsd264rgz44xucnkhpm/bin/g++

[ptrebert]

Ok, it seems that is hard-wired in your default environment. Then simply replace the first entry in the makefile to read as follows:

GPP=/home/sou6/bin/MBG/build_env/bin/x86_64-conda-linux-gnu-gcc

and try again (after make clean)

[oushujun]

Yeah, I tried that. I had the first error:

$ make clean
rm -f obj/*
rm -f bin/*
$ make bin/MBG
cc  -c -o obj/MBG.o src/MBG.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
cc: fatal error: cannot execute 'cc1plus': execvp: No such file or directory
compilation terminated.
make: *** [makefile:30: obj/MBG.o] Error 1

So I install the conda g++ with $ conda install -c conda-forge gxx -y, and tried again:

$ make bin/MBG
/home/sou6/bin/MBG/build_env/bin/x86_64-conda-linux-gnu-gcc  -c -o obj/MBG.o src/MBG.cpp -Wall -Wextra -std=c++17 -O3 -g -Izstr/src -Iparallel-hashmap/parallel_hashmap/ -Wno-unused-parameter -Icxxopts/include -Iconcurrentqueue
In file included from src/fastqloader.h:6,
                 from src/MBG.cpp:13:
zstr/src/zstr.hpp:14:10: fatal error: zlib.h: No such file or directory
   14 | #include <zlib.h>
      |          ^~~~~~~~
compilation terminated.
make: *** [makefile:30: obj/MBG.o] Error 1

# check required zlib file, zstr.hpp:14:10 is missing...
(/home/sou6/bin/MBG/build_env) [sou6@devschatz1 MBG]$ ll zstr/src/
total 98K
-rw-r--r-- 1 sou6 mschatz1 7.6K Feb 18 10:35 strict_fstream.hpp
-rw-r--r-- 1 sou6 mschatz1  15K Feb 18 10:35 zstr.hpp
-rw-r--r-- 1 sou6 mschatz1 2.9K Feb 18 10:35 zstr_make_unique_polyfill.h

# but zlib is already installed...
$ conda install zlib -y -c conda-forge
...
  All requested packages already installed

[maickrau]

Could you try creating a conda environment, installing conda install make gxx_linux-64 zlib pkg-config and then trying to compile in that environment? It shouldn't need the environment for running, just for compiling. Version 1.0.10 is also on bioconda now in case this doesn't work

[oushujun]

@maickrau
Hi Mikko,

Thank you for your suggestions. I followed them and successfully compiled MBG without errors. There are a few warnings and I'll see if they create problems when I get the servers to test it out. I'll keep you posted.

Shujun

[oushujun]

Issue resolved. Thanks!