Issue when trying to run test

[mrmrwinter]

Hi,

I am looking to run Longstitch on my assembly, and need ntlinks installed to do so. I have installed from source code, but when running the test script i receive the following error:

+ ntLink scaffold -B target=scaffolds_1.fa reads=long_reads_1.fa w=250
indexlr --long --pos --strand -k 32 -w 250 -t 4 scaffolds_1.fa > scaffolds_1.fa.k32.w250.tsv
sh -c 'pigz -p4 -f -cd long_reads_1.fa | \
indexlr --long --pos --strand -k 32 -w 250 -t 4 - | \
/home/mike/software/ntLink-1.3.4/bin/ntlink_pair.py -p scaffolds_1.fa.k32.w250.z1000 -n 1 -m scaffolds_1.fa.k32.w250.tsv -s scaffolds_1.fa  \
-k 32 -a 1 -z 1000 -f 10 -x 0  --verbose -'
Running pairing stage of ntLink ...

Parameters:
	Reads TSV files:  ['-']
	-s  scaffolds_1.fa
	-m  scaffolds_1.fa.k32.w250.tsv
	-p  scaffolds_1.fa.k32.w250.z1000
	-n  1
	-k  32
	-a  1
	-z  1000
	-f  10
	-x  0.0
2022-08-03 10:20:20.910853 : Reading minimizers scaffolds_1.fa.k32.w250.tsv
2022-08-03 10:20:20.911557 : Reading fasta file scaffolds_1.fa
2022-08-03 10:20:20.911647 : Finding pairs
2022-08-03 10:20:20.922134 : Building scaffold graph
2022-08-03 10:20:20.922208 : Filtering the graph
2022-08-03 10:20:20.922255 : Printing graph scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot
2022-08-03 10:20:20.922371 : DONE!
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 1 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 2 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 3 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 4 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 5 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 6 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 7 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 8 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 9 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path.sterr'
sh -c 'cat scaffolds_1.fa | \
cut -d " " -f1  | \
abyss-scaffold -k2 -n 10 -s1000 --min-gap 20 - scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot 1> scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path 2>scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path.sterr'
/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py --min_n 1 --max_n 10  -p out \
-g scaffolds_1.fa.k32.w250.z1000.n1.scaffold.dot --conservative scaffolds_1.fa.k32.w250.z1000.n1.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n2.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n3.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n4.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n5.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n6.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n7.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n8.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n9.abyss-scaffold.path scaffolds_1.fa.k32.w250.z1000.n10.abyss-scaffold.path -o scaffolds_1.fa.k32.w250.z1000.stitch.path
Running ntLink stitch paths stage...

2022-08-03 10:20:21.083186  : Finding optimal n...
Traceback (most recent call last):
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 486, in <module>
    main()
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 482, in main
    NtLinkPath().main()
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 426, in main
    best_file = self.find_optimal_n(self.args.PATH)
  File "/home/mike/software/ntLink-1.3.4/bin/ntlink_stitch_paths.py", line 387, in find_optimal_n
    best_n = int(name_match.group(1))
AttributeError: 'NoneType' object has no attribute 'group'
make: *** [/home/mike/software/ntLink-1.3.4/ntLink:203: scaffolds_1.fa.k32.w250.z1000.stitch.path] Error 1

I can't find any mention of this error in relation to ntlinks on Google, so any help would be appreciated.

Many thanks

Edit: i am also getting the same error when trying to run longstitch on my own data

[warrenlr]

hello Michael, how did you install longstitch? If you haven't done so, I'd recommend installing longstitch+dependencies using conda & make a new environment https://anaconda.org/bioconda/longstitch

thanks for your interest in ntlink/longstitch

conda install -c bioconda longstitch

[mrmrwinter]

Hi,

I initially tried to conda install in a fully fresh environment but kept getting an error saying it required python 3.1, whereas the conda install installed 3.10. I tried force installing 3.1 into the environment but it wouldn't resolve. I replicated this error on two machines.

I then tried the brew install method, which failed with a missing ntlinks_bottle_manifest, leading me to try the source code install.

The source install seemingly works, and runs both longstitch and ntlinks tests up until the error I mentioned above

[mrmrwinter]

Accidentally closed, sorry.

[mrmrwinter]

Ive tried again to do the conda install
I ran:
conda create -n longstitch2
conda activate longstitch2
conda install -c bioconda longstitch

But it hung for hours solving the environment:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment:

i then tried installing it using mamba:
'mamba install -c bioconda longstitch`

Which returned:

Looking for: ['longstitch']

bioconda/noarch                                             Using cache
pkgs/main/noarch                                   814.7kB @   2.4MB/s  0.3s
pkgs/r/noarch                                        1.3MB @   2.0MB/s  0.3s
pkgs/r/linux-64                                      1.4MB @   1.4MB/s  0.3s
pkgs/main/linux-64                                   4.6MB @   4.0MB/s  1.2s
bioconda/linux-64                                    4.2MB @   3.5MB/s  1.2s
conda-forge/noarch                                   8.9MB @   4.0MB/s  2.3s
conda-forge/linux-64                                24.5MB @   3.9MB/s  6.4s

Pinned packages:
  - python 3.10.*

Encountered problems while solving:
  - package longstitch-1.0.1-hdfd78af_0 requires ntlink, but none of the providers can be installed

I then tried installing ntlink from conda:
conda install -c bioconda ntlink

Which returned:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.

ResolvePackageNotFound: 
  - python=3.1

I then tried with mamba which returned this:

Looking for: ['ntlink']

bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
pkgs/r/linux-64                                               No change
pkgs/main/noarch                                              No change
pkgs/r/noarch                                                 No change
pkgs/main/linux-64                                            No change

Pinned packages:
  - python 3.10.*

Encountered problems while solving:
  - package ntlink-1.0.1-py39h39abbe0_0 requires python >=3.9,<3.10.0a0, but none of the providers can be installed

I then force installed python 3.9:
conda install python=3.9 -f

This worked, and after this i could succesfully install both ntlink and longstitch using mamba.

I ran longstitch on my assembly and it ran to completion.

[warrenlr]

Thank you for documenting your install attempts.

I am glad to hear of the successful install+execution of longstitch on your system.

For posterity (&other users), would you be able to also record the spec of your system on here?

[mrmrwinter]

Sure.

I'm running on Ubuntu 20.4
Miniconda version 4.10.3

CPU:Ryzen9 3900X
GPU: GTX1660 Super
RAM 64Gb

[lcoombe]

Thanks for the info @mrmrwinter, and I'm glad you got it working!

To confirm, dependencies of LongStitch do require python 3.8 or 3.9 (as you found when installing ntLink separately), which I believe is leading to the initial error you saw with mamba install -c bioconda longstitch. conda install mamba installs python 3.10 by default in tests on my end.

Just to reiterate for any future users who have the issue, if using mamba, either use:

• conda install mamba python=3.9 prior to mamba install -c longstitch
• mamba install -c bioconda python=3.9 longstitch

[Adamtaranto]

Hi @lcoombe, I'm running into a similar issue.

Do you know what parts of ntLink are actually incompatible with Python >= 3.10? Can they easily be upgraded?

[lcoombe]

Hi @Adamtaranto,

Thanks for bringing up this issue again. At least on the ntLink side, that restriction was added by a bioconda maintainer in 2022 - there shouldn't be any ntLink code that is incompatible.
I'll do some tests on my end, and see if updating the ntLink conda recipe itself will do the trick, or if there are other dependencies that cause issues. Will keep you updated!

[Adamtaranto]

I can't see anywhere in the bioconda recipe or in setup.py that is specifically enforcing python <= 3.10. Maybe coming from a dependency?

I also noticed that ntLink can't be installed from bioconda on an M-series mac as there is no osx-arm64 build of the package. You might be able to make the package conda installable on Apple silicon macs by including "noarch: python" in the build section.

I see that the bioconda recipe requires " - {{ compiler('cxx') }}", but it's not clear to me if anything is getting compiled. So might be fine as noarch?

[lcoombe]

Hi @Adamtaranto,

The python3 requirement is in the meta.yaml for the recipe: https://github.com/bioconda/bioconda-recipes/blob/86d1f45fa2731bb8a3fbdb6ef157b38a04254885/recipes/ntlink/meta.yaml#L13
I could see that the line in question was changed by a maintainer using git blame.

Thanks for pointing out the compiler part - you're right, previous versions of ntLink did require compilation, but not since we refactored to make btllib a dependency (vs. packaged in ntLink). I have a busy week, but will aim to open a bioconda PR to update the ntLink recipe this week - I will let you know when I do.

[lcoombe]

I have opened a PR on bioconda (bioconda/bioconda-recipes#45522) to update the recipe to noarch (thus removing the python version restriction). Hopefully it will be merged soon!
Testing the build locally, I found the updates to work fine with python 3.10