Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ig.Graph loading of edge sequences in large graph (18.5 million vertices) produces SystemError #531

Closed
mezwick opened this issue Apr 19, 2022 · 31 comments
Closed

ig.Graph loading of edge sequences in large graph (18.5 million vertices) produces SystemError #531

mezwick opened this issue Apr 19, 2022 · 31 comments

Comments

@mezwick
Copy link

mezwick commented Apr 19, 2022 •
edited

Describe the bug
I am trying to generate an edge weighted graph g from a large anndata adata object (18.5 million cells/rows). I can do this for a subset of data, but when i try to do this for the full dataset i get the following SystemError

#g.es['weight'] = weights
SystemError: /opt/conda/conda-bld/python-split_1649141344976/work/Objects/listobject.c:138: bad argument to internal function

To reproduce
I am trying to generate the graph like so...

import anndata as ad
import scanpy as sc
import igraph as ig

# Read in adata object
adata_path = 'path/to/my/file.h5ad'
adata = ad.read(adata_path)

# Get adjacency atrix
adjacency = sc._utils._choose_graph(adata, obsp=None, neighbors_key=None) # pick the neighbours adjacency matrix

# Get a sparse matrix
sources, targets = adjacency.nonzero() 

# Get weights
weights = adjacency[sources, targets]
weights = weights.A1

# Generate igraph 
g = ig.Graph(directed=None)
g.add_vertices(adjacency.shape[0])  # this adds adjacency.shape[0] vertices
g.add_edges(list(zip(sources, targets)))
g.es['weight'] = weights

Apologies if this is in the wrong place, it may be a problem for the c igraph library.

I have also posted this problem on the scanpy github issues page, though think the issue may be more to do with loading weights into igraph so am bringing it here scverse/scanpy#1053 (comment) .

Version information
conda = 4.10.3
python = 3.9.12
python-igraph = 0.9.10
igraph = 0.9.8
@szhorvat
Copy link
Member

Can you check the values of g.ecount() and g.vcount() just before the g.es['weight'] line and let us know the output?

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

I looked up the source of Python 3.9.12 and it seems like the offending call is a call to PyList_New() with a negative argument. It's completely a stab in the dark yet but I'd suspect this call inside src/_igraph/edgeseqobject.c:

list = PyList_New(no_of_edges);

where no_of_edges is a long. If this is 32-bit and signed, then you are likely to get problems if no_of_edges is larger than or equal to 2^31.

@szhorvat
Copy link
Member

Yes, but 18 million is far from 2^31, so how did we get here?

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

18 million is the number of vertices, not the number of edges.

@szhorvat
Copy link
Member

Ah, right.

@mezwick igraph 0.10 will have overflow checks, and will also use 64-bit integers, so it should be immune to such problems (hopefully!). But with 0.9, the effective limits are at most 2^31 - 1 vertices and 2^30 edges. These values are very likely to cause troubles, and slightly smaller values might too.

@szhorvat
Copy link
Member

If you are willing to compile it, you could try the development version of 0.10 right now. Instructions are here: https://github.com/igraph/python-igraph/#compiling-the-development-version Use the develop branch.

I'm not sure what the state of overflow checks is in the Python glue code, @ntamas ? The C core is in a pretty good shape already, though some stochastic graph generators can still overflow. Anyway, with 64-bit integers overflow will not be practically possible.

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

The Python glue code still uses a mixture of Py_ssize_t, long int and igraph_integer_t and it has to be sorted out eventually, but I'm already doing too many things at the same time so I don't know when I can sort it out properly.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022 •
edited

Thanks for the quick responses. :) So illuminating already! Sounds like i have hit graph size limits.

g.ecount() = -1939291626
how do you interpret a negative ecount?
g.vcount() = 18543700 

adjacency
 = <18543700x18543700 sparse matrix of type '<class 'numpy.float32'>'
	with 2,355,675,670 stored elements in Compressed Sparse Row format>

A 0.9 subset of my data (16,689,330 vertices) is able to run.

So, it would seem the number of edges in the full dataset are too many for the 32 bit integer limit?
I will look into running igraph 0.10 to see if the dev version already works in my case.

Thanks for taking the time to respond.

Best,
Mezwick

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

@mezwick I've briefly checked the code in develop and it seems like we've already made some efforts to make sure that python-igraph works correctly even when you hit the 32-bit limit on the edge counts, but we haven't had the chance to test it thoroughly. Please give it a go with the develop branch and report any issues that you are running into as we are really keen on fixing issues of this kind for the next version.

If you need help with compiling the development branch, let me know.

@szhorvat
Copy link
Member

szhorvat commented Apr 19, 2022 •
edited

how do you interpret a negative ecount?

It's the result of integer overflow. You have more than 2^31 - 1 = 2,147,483,647 edges.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022

Thanks.

I have run...
pip install git+https://github.com/igraph/python-igraph@develop

Is that all i need to do?

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/notebooks# pip install git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
  Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-utwxolmm
  Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-utwxolmm
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin'.
  Resolved https://github.com/igraph/python-igraph to commit dedb177b58463a720fad5bfd5862fa769aea4ac6
  Running command git submodule update --init --recursive -q
Requirement already satisfied: texttable>=1.6.2 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from igraph==0.9.10) (1.6.4)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

It does not appear to have updated the python-igraph package number

Sorry, i suspect i am missing something obvious. Installing the dev branch is new territory for me

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

You might need to run pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph/python-igraph@develop in case pip is trying to be smart and considers the new version being identical to the existing one.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022

i ran the following which took me a little further...
pip install cmake
pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop

but i appear to have hit a wall as need to install flex and bison. These appear to require sudo install permissions, which i don't have. Will look into getting these packages (but think i will not get them easily/quickly)

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install cmake
Collecting cmake
  Downloading cmake-3.22.4-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.5 MB)
     |████████████████████████████████| 22.5 MB 16.0 MB/s 
Installing collected packages: cmake
Successfully installed cmake-3.22.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
  Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-h6yacg3_
  Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-h6yacg3_
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin'.
  Resolved https://github.com/igraph/python-igraph to commit 7c81be69411bfb4cb0f2da7c84489ebf60cb2d0e
  Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
  Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
  Building wheel for igraph (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-47s6j66q
       cwd: /tmp/pip-req-build-h6yacg3_/
  Complete output (145 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
  creating build/lib.linux-x86_64-3.9/igraph/app
  copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
  copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
  creating build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  creating build/lib.linux-x86_64-3.9/igraph/remote
  copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
  copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
  creating build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
  creating build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
  creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  running build_ext
  running build_c_core
  We are going to build the C core of igraph.
    Source folder: vendor/source/igraph
    Build folder: vendor/build/igraph
    Install folder: vendor/install/igraph
  
  Configuring build...
  -- Setting build type to 'Release' as none was specified.
  -- Version number from Git: 0.10.0-dev+c9ed1c6b
  -- The C compiler identification is GNU 7.5.0
  -- The CXX compiler identification is GNU 7.5.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG
  -- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG - Success
  -- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG
  -- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG - Success
  -- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR
  -- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR - Success
  -- Looking for pthread.h
  -- Looking for pthread.h - found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - found
  -- Found Threads: TRUE
  -- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
  -- Could NOT find BISON (missing: BISON_EXECUTABLE)
  -- Looking for stpcpy
  -- Looking for stpcpy - found
  -- Looking for strcasecmp
  -- Looking for strcasecmp - found
  -- Looking for strdup
  -- Looking for strdup - found
  -- Looking for strndup
  -- Looking for strndup - found
  -- Looking for _stricmp
  -- Looking for _stricmp - not found
  CMake Error at src/CMakeLists.txt:24 (bison_target):
    Unknown CMake command "bison_target".
  
  
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
  See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeError.log".
  Build failed for the C core of igraph.
  
  ----------------------------------------
  ERROR: Failed building wheel for igraph
  Running setup.py clean for igraph
Failed to build igraph
Installing collected packages: texttable, igraph
  Attempting uninstall: texttable
    Found existing installation: texttable 1.6.4
    Uninstalling texttable-1.6.4:
      Successfully uninstalled texttable-1.6.4
  Attempting uninstall: igraph
    Found existing installation: igraph 0.9.10
    Uninstalling igraph-0.9.10:
      Successfully uninstalled igraph-0.9.10
    Running setup.py install for igraph ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zix7h65k/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph
         cwd: /tmp/pip-req-build-h6yacg3_/
    Complete output (109 lines):
    running install
    /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
    creating build/lib.linux-x86_64-3.9/igraph/app
    copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
    copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
    creating build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    creating build/lib.linux-x86_64-3.9/igraph/remote
    copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
    copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
    creating build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
    creating build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
    creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    running build_ext
    running build_c_core
    We are going to build the C core of igraph.
      Source folder: vendor/source/igraph
      Build folder: vendor/build/igraph
      Install folder: vendor/install/igraph
    
    Configuring build...
    -- Version number from Git: 0.10.0-dev+c9ed1c6b
    -- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
    -- Could NOT find BISON (missing: BISON_EXECUTABLE)
    CMake Error at src/CMakeLists.txt:24 (bison_target):
      Unknown CMake command "bison_target".
    
    
    -- Configuring incomplete, errors occurred!
    See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
    See also "/tmp/pip-req-build-h6yacg3_/vendor/build/igraph/CMakeFiles/CMakeError.log".
    Build failed for the C core of igraph.
    
    ----------------------------------------
  Rolling back uninstall of igraph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/igraph
   from /tmp/pip-uninstall-t33v86fj/igraph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/~graph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph-0.9.10.dist-info/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph-0.9.10.dist-info
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph
ERROR: Command errored out with exit status 1: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h6yacg3_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zix7h65k/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph Check the logs for full command output.

@mezwick mezwick mentioned this issue Apr 19, 2022
Closed
@szhorvat
Copy link
Member

If you don't have root access, you should still be able to install flex and bison with conda.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022 •
edited

i did try pip install of flex and bison in my conda-env, but appear to get the same error...?

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install bison
Collecting bison
  Downloading bison-0.1.3-py2.py3-none-any.whl (24 kB)
Collecting pyyaml>=5.4
  Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)
     |████████████████████████████████| 661 kB 33.0 MB/s 
Installing collected packages: pyyaml, bison
Successfully installed bison-0.1.3 pyyaml-6.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install flex
Collecting flex
  Downloading flex-6.14.1.tar.gz (40 kB)
     |████████████████████████████████| 40 kB 4.3 MB/s 
Requirement already satisfied: six>=1.7.3 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (1.16.0)
Requirement already satisfied: PyYAML>=3.11 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (6.0)
Collecting validate-email>=1.2
  Downloading validate_email-1.3.tar.gz (4.7 kB)
Collecting rfc3987>=1.3.4
  Downloading rfc3987-1.3.8-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: requests>=2.4.3 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from flex) (2.27.1)
Collecting strict-rfc3339>=0.7
  Downloading strict-rfc3339-0.7.tar.gz (17 kB)
Collecting click>=3.3
  Downloading click-8.1.2-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 6.4 MB/s 
Collecting jsonpointer>=1.7
  Downloading jsonpointer-2.3-py2.py3-none-any.whl (7.8 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (1.26.9)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages (from requests>=2.4.3->flex) (3.3)
Building wheels for collected packages: flex, strict-rfc3339, validate-email
  Building wheel for flex (setup.py) ... done
  Created wheel for flex: filename=flex-6.14.1-py3-none-any.whl size=77276 sha256=ce28b92100fcdd97b5cfef49b531df243f1779830a0fa640ff61acf565df5514
  Stored in directory: /home/dnanexus/.cache/pip/wheels/e9/9a/ef/ea72c07b1be4b3dfebba94d22f905ff8ad53be48419e6114d1
  Building wheel for strict-rfc3339 (setup.py) ... done
  Created wheel for strict-rfc3339: filename=strict_rfc3339-0.7-py3-none-any.whl size=18149 sha256=33106798d06e29316fdd30084a2f5766e3ae27448d79e03f7a12c91ae435ad38
  Stored in directory: /home/dnanexus/.cache/pip/wheels/25/38/74/7ec7f77ec64b2907430120931ba588b40e6e26f02d4df5be35
  Building wheel for validate-email (setup.py) ... done
  Created wheel for validate-email: filename=validate_email-1.3-py3-none-any.whl size=5482 sha256=fb788ddd7e3faf3a923b38a61bea2c01518db8d62513c0859aff2426542f7891
  Stored in directory: /home/dnanexus/.cache/pip/wheels/84/19/e3/329ee20a29b195fa193ee3af2475fa9392b08dde8ba893c714
Successfully built flex strict-rfc3339 validate-email
Installing collected packages: validate-email, strict-rfc3339, rfc3987, jsonpointer, click, flex
Successfully installed click-8.1.2 flex-6.14.1 jsonpointer-2.3 rfc3987-1.3.8 strict-rfc3339-0.7 validate-email-1.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
  Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-91k96vz1
  Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-91k96vz1
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin'.
  Resolved https://github.com/igraph/python-igraph to commit 7c81be69411bfb4cb0f2da7c84489ebf60cb2d0e
  Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
  Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
  Building wheel for igraph (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-cckilrhh
       cwd: /tmp/pip-req-build-91k96vz1/
  Complete output (145 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
  copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
  creating build/lib.linux-x86_64-3.9/igraph/app
  copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
  copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
  creating build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
  creating build/lib.linux-x86_64-3.9/igraph/remote
  copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
  copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
  creating build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
  copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
  creating build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
  copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
  creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
  creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
  creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
  running build_ext
  running build_c_core
  We are going to build the C core of igraph.
    Source folder: vendor/source/igraph
    Build folder: vendor/build/igraph
    Install folder: vendor/install/igraph
  
  Configuring build...
  -- Setting build type to 'Release' as none was specified.
  -- Version number from Git: 0.10.0-dev+c9ed1c6b
  -- The C compiler identification is GNU 7.5.0
  -- The CXX compiler identification is GNU 7.5.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG
  -- Performing Test COMPILER_SUPPORTS_NO_VARARGS_FLAG - Success
  -- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG
  -- Performing Test COMPILER_SUPPORTS_NO_UNKNOWN_WARNING_OPTION_FLAG - Success
  -- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR
  -- Performing Test COMPILER_HAS_DEPRECATED_ENUMVAL_ATTR - Success
  -- Looking for pthread.h
  -- Looking for pthread.h - found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - found
  -- Found Threads: TRUE
  -- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
  -- Could NOT find BISON (missing: BISON_EXECUTABLE)
  -- Looking for stpcpy
  -- Looking for stpcpy - found
  -- Looking for strcasecmp
  -- Looking for strcasecmp - found
  -- Looking for strdup
  -- Looking for strdup - found
  -- Looking for strndup
  -- Looking for strndup - found
  -- Looking for _stricmp
  -- Looking for _stricmp - not found
  CMake Error at src/CMakeLists.txt:24 (bison_target):
    Unknown CMake command "bison_target".
  
  
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
  See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeError.log".
  Build failed for the C core of igraph.
  
  ----------------------------------------
  ERROR: Failed building wheel for igraph
  Running setup.py clean for igraph
Failed to build igraph
Installing collected packages: texttable, igraph
  Attempting uninstall: texttable
    Found existing installation: texttable 1.6.4
    Uninstalling texttable-1.6.4:
      Successfully uninstalled texttable-1.6.4
  Attempting uninstall: igraph
    Found existing installation: igraph 0.9.10
    Uninstalling igraph-0.9.10:
      Successfully uninstalled igraph-0.9.10
    Running setup.py install for igraph ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_596473v/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph
         cwd: /tmp/pip-req-build-91k96vz1/
    Complete output (109 lines):
    running install
    /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/matching.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/sparse_matrix.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/summary.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/version.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/__init__.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/adjacency.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/automorphisms.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/basic.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/bipartite.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/clustering.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/community.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/configuration.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/cut.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/datatypes.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/formula.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/layout.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/seq.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/statistics.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/structural.py -> build/lib.linux-x86_64-3.9/igraph
    copying src/igraph/utils.py -> build/lib.linux-x86_64-3.9/igraph
    creating build/lib.linux-x86_64-3.9/igraph/app
    copying src/igraph/app/__init__.py -> build/lib.linux-x86_64-3.9/igraph/app
    copying src/igraph/app/shell.py -> build/lib.linux-x86_64-3.9/igraph/app
    creating build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/metamagic.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/baseclasses.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/colors.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/shapes.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    copying src/igraph/drawing/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing
    creating build/lib.linux-x86_64-3.9/igraph/remote
    copying src/igraph/remote/__init__.py -> build/lib.linux-x86_64-3.9/igraph/remote
    copying src/igraph/remote/gephi.py -> build/lib.linux-x86_64-3.9/igraph/remote
    creating build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/__init__.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/adjacency.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/bipartite.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/files.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/images.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/libraries.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/objects.py -> build/lib.linux-x86_64-3.9/igraph/io
    copying src/igraph/io/random.py -> build/lib.linux-x86_64-3.9/igraph/io
    creating build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/__init__.py -> build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/functions.py -> build/lib.linux-x86_64-3.9/igraph/operators
    copying src/igraph/operators/methods.py -> build/lib.linux-x86_64-3.9/igraph/operators
    creating build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/base.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/coord.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/plot.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/text.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    copying src/igraph/drawing/cairo/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/cairo
    creating build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/dendrogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/histogram.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/matrix.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/palette.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    copying src/igraph/drawing/matplotlib/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/matplotlib
    creating build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/__init__.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/edge.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/graph.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/polygon.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/utils.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    copying src/igraph/drawing/plotly/vertex.py -> build/lib.linux-x86_64-3.9/igraph/drawing/plotly
    running build_ext
    running build_c_core
    We are going to build the C core of igraph.
      Source folder: vendor/source/igraph
      Build folder: vendor/build/igraph
      Install folder: vendor/install/igraph
    
    Configuring build...
    -- Version number from Git: 0.10.0-dev+c9ed1c6b
    -- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
    -- Could NOT find BISON (missing: BISON_EXECUTABLE)
    CMake Error at src/CMakeLists.txt:24 (bison_target):
      Unknown CMake command "bison_target".
    
    
    -- Configuring incomplete, errors occurred!
    See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeOutput.log".
    See also "/tmp/pip-req-build-91k96vz1/vendor/build/igraph/CMakeFiles/CMakeError.log".
    Build failed for the C core of igraph.
    
    ----------------------------------------
  Rolling back uninstall of igraph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/igraph
   from /tmp/pip-uninstall-insy6qii/igraph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/~graph
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph-0.9.10.dist-info/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph-0.9.10.dist-info
  Moving to /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/igraph/
   from /opt/conda/envs/scanpy_py3pt9_igraph0pt10/lib/python3.9/site-packages/~graph
ERROR: Command errored out with exit status 1: /opt/conda/envs/scanpy_py3pt9_igraph0pt10/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-91k96vz1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_596473v/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/scanpy_py3pt9_igraph0pt10/include/python3.9/igraph Check the logs for full command output.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022

flex and bison via conda install seems to be working...

@szhorvat
Copy link
Member

These appear to require sudo install permissions, which i don't have.

Why is pip complaining that you are running as root if you do not have root access? See:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022

Ok. So it seems i do have sudo permissions, I just needed to run
apt update && apt upgrade
and
apt install sudo
Apologies for the confusion.

I tried

You might need to run pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph/python-igraph@develop in case pip is trying to be smart and considers the new version being identical to the existing one.

but got the following

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@develop Collecting git+https://github.com/igraph
/python-igraph@develop
Collecting git+https://github.com/igraph/python-igraph@develop
  Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-7kkjy0vx
  Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-7kkjy0vx
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin'.
  Resolved https://github.com/igraph/python-igraph to commit 64a9df3f768ba62346e99d397ef9181a56317e27
  Running command git submodule update --init --recursive -q
ERROR: Could not find a version that satisfies the requirement Collecting (from versions: none)
ERROR: No matching distribution found for Collecting

So instead ran,

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# pip install --upgrade --force-reinstall git+https://github.com/igraph/python-igraph@developCollecting git+https://github.com/igraph/python-igraph@develop
  Cloning https://github.com/igraph/python-igraph (to revision develop) to /tmp/pip-req-build-gpm2j_q9
  Running command git clone -q https://github.com/igraph/python-igraph /tmp/pip-req-build-gpm2j_q9
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin'.
  Resolved https://github.com/igraph/python-igraph to commit 64a9df3f768ba62346e99d397ef9181a56317e27
  Running command git submodule update --init --recursive -q
Collecting texttable>=1.6.2
  Using cached texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: igraph
  Building wheel for igraph (setup.py) ... done
  Created wheel for igraph: filename=igraph-0.9.10-cp39-cp39-linux_x86_64.whl size=2076664 sha256=1d5da72fd5c86a7dbe172eb04c8e10e22542948c39e50d1edaf5498e337ad4c7
  Stored in directory: /tmp/pip-ephem-wheel-cache-dz_rhlmk/wheels/fd/36/d1/e31f410d36dbdef5d45386d89324b9fc71fc00d1dd7313c3b7
Successfully built igraph
Installing collected packages: texttable, igraph
  Attempting uninstall: texttable
    Found existing installation: texttable 1.6.4
    Uninstalling texttable-1.6.4:
      Successfully uninstalled texttable-1.6.4
  Attempting uninstall: igraph
    Found existing installation: igraph 0.9.10
    Uninstalling igraph-0.9.10:
      Successfully uninstalled igraph-0.9.10
Successfully installed igraph-0.9.10 texttable-1.6.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph#

However, it still seems my igraph and python-graph version numbers have not updated?

(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph# conda list igraph
# packages in environment at /opt/conda/envs/scanpy_py3pt9_igraph0pt10:
#
# Name                    Version                   Build  Channel
igraph                    0.9.8                hf5496dd_0    conda-forge
python-igraph             0.9.10           py39hf6748bb_0    conda-forge
(scanpy_py3pt9_igraph0pt10) root@dcd19bf51c7b:/opt/git/python-igraph#

@ntamas
Copy link
Member

ntamas commented Apr 19, 2022

I think the compilation went fine; you will not see an updated version number in python-igraph because you are on the develop branch and no new version has been assigned to the develop branch yet. The version number is taken from src/igraph/version.py, which still shows 0.9.10 on the development branch.

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022 •
edited

fantastic. thanks for your help both.

i have hit run on my large graph script in the new environment.... will let you know if it runs!

@iosonofabio
Copy link
Member

Happy to help fix the 32 -> 64 bit conversion for this if needed, since I know that glue code decently and @ntamas might be too busy.

But let's see if it's necessary at all

@mezwick
Copy link
Author

mezwick commented Apr 19, 2022

64-bit version of igraph looks to still be running fine so far.

from my log...
my script has 2,355,675,670 edges (no longer a negative edge count)
edge weights were calculated and added to the graph just fine,
igraph is now proceeding to leiden cluster (with igraph g.community_leiden())....

2022-04-19 16:31:04 21197 INFO     adata read finished 
2022-04-19 16:31:04 21197 INFO     adata full length, 18,543,700 
2022-04-19 16:31:04 21197 INFO     adjacency extracted 
2022-04-19 16:31:52 21197 INFO     adjacency sources, targets extracted 
2022-04-19 16:32:59 21197 INFO     weights calculated 
2022-04-19 16:32:59 21197 INFO     graph initiated 
2022-04-19 16:33:00 21197 INFO     graph vertices added 
2022-04-19 16:33:00 21197 INFO     vcount: 18543700 
2022-04-19 17:18:52 21197 INFO     graph edges added 
2022-04-19 17:18:52 21197 INFO     ecount: 2355675670 
2022-04-19 17:21:24 21197 INFO     graph weights added 
2022-04-19 17:21:24 21197 INFO     leiden clustering begun with 0.1 resolution

@iosonofabio
Copy link
Member

Sounds promising! Maybe my in64 PR wasn't totally broken.

Btw we'll have UMAP in igraph within a day or two, so if you want to embed the graph you can use igraph for both clustering and embedding.

@szhorvat
Copy link
Member

szhorvat commented Apr 20, 2022 •
edited

I'll close this now, since this is effectively fixed on develop (and the issue itself has turned into a support thread).

@ntamas
Copy link
Member

ntamas commented Apr 20, 2022

Well, at least the loading part is fixed. @mezwick if you find any issues with processing a graph this large, please file another issue with the details

@mezwick
Copy link
Author

mezwick commented Apr 20, 2022

Thankyou all for your help. I appreciate your time.

@mezwick
Copy link
Author

mezwick commented Apr 20, 2022

happy to say g.community_leiden() runs on my 64bit igraph with 18.5 million cells :) @vtraag

@szhorvat
Copy link
Member

How long did it take to complete?

@mezwick
Copy link
Author

mezwick commented Apr 20, 2022 •
edited

Leiden run time: 5h 26 mins 23.798647 secs
On a cluster with 3892 GB mem, 16384 GB storage, 128 cores.
PID took ~9% of memory... so ~350 GB RAM needed
g.community_leiden() just seems to run on a single core (100 % CPU in top)

@vtraag
Copy link
Member

vtraag commented Apr 20, 2022

Great, good to hear you managed to run everything @mezwick! Sounds like quite a reasonable run time. Sorry for my late reply, I only get to reading this now. Thanks all for helping out!

@ntamas
Copy link
Member

ntamas commented Apr 20, 2022

g.community_leiden() just seems to run on a single core (100 % CPU in top)

Yes, that's correct; igraph is currently not guaranteed to be thread-safe and the algorithms use single-core implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ntamas @iosonofabio @szhorvat @vtraag @mezwick