Error: "could not convert string to float"

[Xiaofei-git]

I "pip3 install mikado" installed the mikado on our hpc cluster and used the "sample_data" to test it from command line. I got same error as "#164". Could you direct me what is the issue?

There is warning message at the beginning "calling yaml.load() without Loader=... is deprecated", might it be a problem for the error? Also, I need to copy "plants.yaml" manually to my working directory. It might be a problem, right? Could you help me figure it out?

Thanks a lot!

Xiaofei

$ mikado configure --list list.txt --reference chr5.fas --mode permissive --scoring plants.yaml -t 8 \

--copy-scoring plants.yaml --junctions junctions.bed -bt uniprot_sprot_plants.fasta configuration.yaml

..../.local/lib/python3.6/site-packages/Mikado/configuration/configurator.py:529: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
scoring = yaml.load(scoring_file)
2019-09-23 14:30:27,523 - main - init.py:124 - ERROR - main - MainProcess - Mikado crashed, cause:
2019-09-23 14:30:27,523 - main - init.py:125 - ERROR - main - MainProcess - Malformed inputs file. Error:
could not convert string to float:
Traceback (most recent call last):
File ".../.local/lib/python3.6/site-packages/Mikado/subprograms/configure.py", line 217, in create_config
score = float(_fields[3])
ValueError: could not convert string to float:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File ".../.local/lib/python3.6/site-packages/Mikado/init.py", line 110, in main
args.func(args)
File ".../.local/lib/python3.6/site-packages/Mikado/subprograms/configure.py", line 222, in create_config
raise ValueError("Malformed inputs file. Error:\n{}".format(exc))
ValueError: Malformed inputs file. Error:
could not convert string to float:

[lucventurini]

Dear @xiaofei,
Thank you for reporting the bug.
Regarding the yaml error message, it is due to updates in the module. I fixed it in the most recent versions of the tool, but older versions are unfortunately bound to have it. It should not affect the results, though.

As for the main error, I think I changed the format of the list. May I ask you which version have you installed? It should be 1.2, correct?

[Xiaofei-git]

Yes, it is v1.2.4.
I used "pip3 install mikado" to install, and thought it is the latest version by the installation. How can I update the tool?
Thanks a lot!

[lucventurini]

Dear @Xiaofei-git , I am about to release a new release candidate this afternoon. That version has the bug fixed (together with many other enhancements).

Best

[Xiaofei-git]

Dear @Xiaofei-git , I am about to release a new release candidate this afternoon. That version has the bug fixed (together with many other enhancements).

Best

Let me know when you finish the release. Thank you so much!

BEST,

Xiaofei

[lucventurini]

Dear @Xiaofei-git , the new release can be found here:

https://github.com/EI-CoreBioinformatics/mikado/releases/tag/2.0rc5

Best

[Xiaofei-git]

Dear @Xiaofei-git , the new release can be found here:

https://github.com/EI-CoreBioinformatics/mikado/releases/tag/2.0rc5

Best

If I'd like to install by "pip3 install mikado", how could I install the new release one? Thanks a lot!
BEST

[lucventurini]

Dear @Xiaofei-git, I added to the release page two binary packages (the ones that finish with .whl), one for python 3.6 and another for python 3.7. Either should function by doing a

pip install mikado.whl.

If you have a different version of python or you are not on a Linux system, then please download the source package (either the tar.gz or the .zip file), unpack them, then

python setup.py bdist_wheel.

You might need to install wheel, if it complains, with pip.
This will create the proper binary package in the dist/ folder, which can be installed with pip:

pip install dist/*whl

I hope this helps.

Best

[Xiaofei-git]

Dear @Xiaofei-git, I added to the release page two binary packages (the ones that finish with .whl), one for python 3.6 and another for python 3.7. Either should function by doing a

pip install mikado.whl.

If you have a different version of python or you are not on a Linux system, then please download the source package (either the tar.gz or the .zip file), unpack them, then

python setup.py bdist_wheel.

You might need to install wheel, if it complains, with pip.
This will create the proper binary package in the dist/ folder, which can be installed with pip:

pip install dist/*whl

I hope this helps.

Best

I used "pip install --user Mikado-2.0rc5-cp36-cp36m-linux_x86_64.whl" to install it, it gave me error but showed me "successfully installed" in the end as below, could I ignor the error and move forward to use it?

Thanks a lot!

ERROR: pygenometracks 2.1 requires hicexplorer>=2.1.1, which is not installed.
ERROR: hicmatrix 7 has requirement intervaltree==2.1.*, but you'll have intervaltree 3.0.2 which is incompatible.
Installing collected packages: numpy, python-rapidjson, msgpack, jsonref, fastnumbers, rapidjson, Mikado
Found existing installation: Mikado 1.2.4
Uninstalling Mikado-1.2.4:
Successfully uninstalled Mikado-1.2.4
Successfully installed Mikado-2.0rc5 fastnumbers-2.2.1 jsonref-0.2 msgpack-0.6.2 numpy-1.17.2 python-rapidjson-0.8.0 rapidjson-1.0.0

[lucventurini]

Dear @Xiaofei-git ,
the issues that pip is complaining about regard two packages that have nothing to do with Mikado, ie pygenometracks and hicmatrix. The installation of Mikado itself should be unaffected.
Please let me know how it goes.

[Xiaofei-git]

I am moving forward on the sample_data.
When I run "makeblastdb", I got error as below, although "mikado.blast.xml.gz" was still generated, but I do't know if it does matter? I googled the error, it might be with the version of blast. Do you have any ideas?

Then, I still move forward with the generated "mikado.blast.xml.gz" for "mikado serialise". Unfortunately, another error as below, I did't get any help by googling, do you have any ideas? Also, am I right for the value of "--blast_targets" option?

So, I asked 3 questions above, hope I didn't mess it up. Thanks a lot!

$ makeblastdb -in uniprot_sprot_plants.fasta -dbtype prot -parse_seqids > blast_prepare.log
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Title is very long: 1001 characters (max is 1000)
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.
Error: (1431.1) FASTA-Reader: Warning: FASTA-Reader: Ignoring FASTA modifier(s) found because the input was not expected to have any.

$ mikado serialise --json-conf configuration.yaml --xml mikado.blast.xml.gz --orfs mikado.bed --blast_targets uniprot_sprot_plants.fasta
./serialise.log
2019-10-01 12:25:58,113 - main - init.py:120 - ERROR - main - MainProcess - Mikado crashed, cause:
2019-10-01 12:25:58,113 - main - init.py:121 - ERROR - main - MainProcess - (sqlite3.OperationalError) disk I/O error
[SQL: CREATE TABLE DB(id int);]
(Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File ".local/lib/python3.6/site-packages/Mikado/init.py", line 106, in main
args.func(args)
File ".local/lib/python3.6/site-packages/Mikado/subprograms/serialise.py", line 379, in serialise
load_orfs(args, logger)
File ".local/lib/python3.6/site-packages/Mikado/subprograms/serialise.py", line 147, in load_orfs
logger=logger)
File ".local/lib/python3.6/site-packages/Mikado/serializers/orf.py", line 222, in init
self.engine = connect(json_conf, logger)
File ".local/lib/python3.6/site-packages/Mikado/utilities/dbutils.py", line 128, in connect
DBBASE.metadata.create_all(engine, checkfirst=True)
File ".local/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4294, in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2035, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/sonas-hs/it/hpc/home/easybuild/install_prod/software/MPI/GCC/7.3.0-2.30/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2027, in _optional_conn_ctx_manager
with self._contextual_connect() as conn:
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2229, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2265, in _wrap_pool_connect
return fn()
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 361, in connect
return _ConnectionFairy._checkout(self, self._fairy)
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 344, in _do_get
c = self._create_connection()
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in init
self.__connect(first_connect_check=True)
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File ".local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 249, in
return lambda crec: creator()
File ".local/lib/python3.6/site-packages/Mikado/utilities/dbutils.py", line 55, in create_connector
create_database("sqlite:///{}".format(db_settings["db"]))
File ".local/lib/python3.6/site-packages/sqlalchemy_utils/functions/database.py", line 584, in create_database
engine.execute("CREATE TABLE DB(id int);")
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2169, in execute
return connection.execute(statement, *multiparams, **params)
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 982, in execute
return self.execute_text(object, multiparams, params)
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text
parameters,
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File ".local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File ".local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File ".local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[SQL: CREATE TABLE DB(id int);]
(Background on this error at: http://sqlalche.me/e/e3q8)

[lucventurini]

Dear @Xiaofei-git ,

regarding the first error (in makeblastdb) - this is due to some sequence titles being too long. It should not pose an issue, in my experience.

Regarding the second issue: unfortunately I think this has to do with the specifics of your system rather than with Mikado. The problem is that the SQLite database used by Mikado crashes. My suspicion is that you are working on a NFS filesystem, which will cause SQLite to fail (I am having a similar issue on some of my machines as well). I would check with your sysadmin.

I hope this helps.

[Xiaofei-git]

Dear @Xiaofei-git ,

regarding the first error (in makeblastdb) - this is due to some sequence titles being too long. It should not pose an issue, in my experience.

Regarding the second issue: unfortunately I think this has to do with the specifics of your system rather than with Mikado. The problem is that the SQLite database used by Mikado crashes. My suspicion is that you are working on a NFS filesystem, which will cause SQLite to fail (I am having a similar issue on some of my machines as well). I would check with your sysadmin.

I hope this helps.

Dear @lucventurini,
I got the error about SQLite, because our home directories are out of space. Now, it is fixed.

[lucventurini]

Dear @Xiaofei-git, thank you for updating us. Glad it worked out in the end.

[Xiaofei-git]

However, when I run the code "$ mikado serialise --json-conf configuration.yaml --xml mikado.blast.xml.gz --orfs mikado.bed --blast_targets uniprot_sprot_plants.fasta", I got the error below, which complained that the provided ORFs do not match the transcripts provided and already present in the database.
The "mikado.bed" file is in the path of "sample_data". How did you get it? In another words, how can I get the bed file for my own data? Run TransDecoder on "mikado_prepared.fasta", which was generated by "mikado prepare"?

Actually, if I don't provide "--orfs mikado.bed", it will go through successfully, just skip it for loading ORFs, "2019-10-10 14:27:41,768 - serialiser - serialise.py:155 - INFO - load_orfs - MainProcess - No ORF data provided, skipping". Why does it complain the provided ORFs do not match the transcripts provided when I used "--orfs mikado.bed"?

Another question is how should I set the value for '--blast_targets'? If I work on Sorghum, it should be fine to use "--blast_targets uniprot_sprot_plants.fasta", right? Thanks a lot!

$ more serialise.log
2019-10-10 13:51:43,460 - serialiser - serialise.py:290 - INFO - setup - MainProcess - Command line: /sonas-hs/ware/hpc/home/xwang/.local/bin
/mikado serialise --json-conf configuration.yaml --xml mikado.blast.xml.gz --orfs mikado.bed --blast_targets uniprot_sprot_plants.fasta
2019-10-10 13:51:43,460 - serialiser - serialise.py:296 - INFO - setup - MainProcess - Random seed: 4170354093
2019-10-10 13:51:43,462 - serialiser - serialise.py:334 - INFO - setup - MainProcess - Using a sqlite database (location: /mnt/grid/ware/hpc/
home/data/xwang/software/mikado_2.0/mikado-2.0rc5/sample_data/mikado.db)
2019-10-10 13:51:43,462 - serialiser - serialise.py:338 - INFO - setup - MainProcess - Requested 4 threads, forcing single thread: False
2019-10-10 13:51:43,463 - serialiser - serialise.py:141 - INFO - load_orfs - MainProcess - Starting to load ORF data
2019-10-10 13:51:43,626 - serialiser - orf.py:402 - CRITICAL - __serialize_multiple_threads - MainProcess - The provided ORFs do not match th
e transcripts provided and already present in the database.Please check your input files.
2019-10-10 13:51:43,627 - serialiser - serialise.py:150 - CRITICAL - load_orfs - MainProcess - Mikado serialise failed due to problems with t
he input data. Please check the logs.

[lucventurini]

The "mikado.bed" file is in the path of "sample_data". How did you get it? In another words, how can I get the bed file for my own data? Run TransDecoder on "mikado_prepared.fasta", which was generated by "mikado prepare"?

Yes. Transdecoder or prodigal (or any other orf caller) must be run on the output of mikado prepare, not of the input transcript models. I thought I had clarified that on the error message, but I will make it more explicit. Thank you for pointing it out.

[Xiaofei-git]

I thought I had clarified that on the error message, but I will make it more explicit. Thank you for pointing it out.

I am sorry I feel confused. I don't think I asked the error before about "The provided ORFs do not match th
e transcripts provided", did I? I also tried to googled it, but I don't find anything related to it.

[lucventurini]

I thought I had clarified that on the error message, but I will make it more explicit. Thank you for pointing it out.

I am sorry I feel confused. I don't think I asked the error before about "The provided ORFs do not match th
e transcripts provided", did I? I also tried to googled it, but I don't find anything related to it.

That's our fault in not making the documentation clear enough. You are not the first person to make this mistake.
I will clarify this in the documentation (#136), and have mikado both have a more informative error in serialise and print it out as an instruction at the end of prepare.

As I mentioned, you are not the first one to make this mistake, so it is clearly on us to clarify the procedure.

[Xiaofei-git]

Thank you so much! Now, it is clear to me and I run through mikado successfully. Here is the code I used for "serialise" step as below. FYI, #114 helped me too, because I had to delete the mikado.db file before re-running Mikado serialise, otherwise it will report that
2019-10-11 11:08:24,220 - main - init.py:120 - ERROR - main - MainProcess - Mikado crashed, cause:
2019-10-11 11:08:24,220 - main - init.py:121 - ERROR - main - MainProcess - I should have serialised 124 ORFs, but 248 are present!

$ mikado serialise --json-conf configuration.yaml --xml mikado.blast.xml.gz --orfs mikado_prepared.fasta.transdecoder.bed --blast_targets uniprot_sprot_plants.fasta -–transcripts mikado_prepared.fasta

BTW, I think it might be better to update the document for serialise here https://mikado.readthedocs.io/en/latest/Tutorial/index.html#creating-the-configuration-file-for-mikado too, because I run through the manual to get familiar with Mikado.

Thank you so much for your help to walk through the errors I met!

[lucventurini]

No problem @Xiaofei-git , very glad I was able to assist you through this! Please let us know if you encounter any other issue.

FYI, #114 helped me too, because I had to delete the mikado.db file before re-running Mikado serialise, otherwise it will report that [..]

Yes, I can understand that. I probably should delete the database when mikado serialise errors as badly as it does when trying to use the wrong ORFs.

BTW, I think it might be better to update the document for serialise here https://mikado.readthedocs.io/en/latest/Tutorial/index.html#creating-the-configuration-file-for-mikado too, because I run through the manual to get familiar with Mikado.

Working on that, thank you!