ffindex_order issue and observation

[fslee62]

hello,
i am using a hh-suite v3 version from oct 2016. not sure if it matters for this following issue.
to optimize the a3m and hhm ffindices, i followed the instructions on page 17 of the manual using ffindex_order. the file "sorting.dat" only worked for the a3m case, but not the hhm case.

so i made a guess replacing all the a3m extensions within the "sorting.dat" file with hhm extensions. using this new file (refer to here as sorting.hhm), i was able to get the hhm case to WORK. that is, it needed 2 separate "sorting.dat" files to optimize the a3m and hhm cases. is this an issue of the manul or something else?

in addition, i noticed that the number of entries before and after the optimization are the same for the a3m case but NOT the same for the hhm case using my trial procedure described above. after optimization, i had fewer hhm entries. is this ok or something is wrong?

-rw-r----- 1 fslee lee 3084984 Jul 5 16:39 test_cs219.ffdata
-rw-r----- 1 fslee lee 391634 Jul 5 16:39 test_cs219.ffindex
-rw-r----- 1 fslee lee 6552214670 Jul 5 16:40 test_a3m.ffdata
-rw-r----- 1 fslee lee 493334 Jul 5 16:40 test_a3m.ffindex
-rw-r----- 1 fslee lee 541748872 Jul 5 16:48 test_hhm.ffdata
-rw-r----- 1 fslee lee 467672 Jul 5 16:48 test_hhm.ffindex
-rw-r----- 1 fslee lee 200868 Jul 5 16:52 sorting.a3m
-rw-r----- 1 fslee lee 6552214670 Jul 5 17:03 test_a3m_new.ffdata
-rw-r----- 1 fslee lee 493334 Jul 5 17:03 test_a3m_new.ffindex
-rw-r----- 1 fslee lee 200868 Jul 5 17:05 sorting.hhm
-rw-r----- 1 fslee lee 430943602 Jul 5 17:36 test_hhm_new.ffdata
-rw-r----- 1 fslee lee 367354 Jul 5 17:36 test_hhm_new.ffindex
muon:/tmp/scope70_2.06.191 wc -l sorting.*
16739 sorting.a3m
16739 sorting.hhm
33478 total
muon:/tmp/scope70_2.06.192 wc -l *.ffindex
16739 test_a3m.ffindex
16739 test_a3m_new.ffindex
16739 test_cs219.ffindex
16861 test_hhm.ffindex
13400 test_hhm_new.ffindex
80478 total

fred

[milot-mirdita]

Could you post the first few lines of each file:

Something like:

head -n 10 test_{a3m,hhm}{,_new}.ff{data,index} test_cs219.ffindex sorting*

Also please update to the latest git version (including git submodules).

[fslee62]

thanks much for the quick reply. here are the file heads:

heads.txt

by the way, i was using downloaded a3m and hhm files from SCOPe2.06 as a test case.

[fslee62]

update: i just learned today that my "work-around" described at the beginning no longer work. in short, i can use ffindex_order to optimize the a3m.{ffdata,ffindex}, but NOT the hhm.{ffdata,ffindex}. here is the typical error message:

~/triad/tools/hhsuite/hhsuite-latest/bin/ffindex_order sorting_hhm test_hhm.ffdata test_hhm.ffindex test_hhm_new.ffdata test_hhm_new.ffindex
ffindex_order: Problem with data file. Is the file empty or is another process reading it?: Invalid argument
ffindex_index_parse failed: Invalid argument

[fslee62]

by the way, i was using a git clone from today (6jul2017). still seeing the same issue.

[milot-mirdita]

For your new issue, it sounds like you have overwritten the test_hhm.ffindex with something else.

Could you check that the databases are still complete and then either upload the *_{a3m,hhm,cs219}.ff{data,index} files somewhere and link them here, or if the data is very sensitive, please send me them to me directly at milot at mirdita dot de

Best regards,
Milot

[fslee62]

hi milot,
i am attaching a small directory called test. it has 3 sets of a3m and hhm files. the file prefixes are the pdb IDs. i was using them to test the making and optimizing of the hhsuite v3 databases. the log file shows the error. the file "sorting_a3m" is the "sorting.dat" in the user guide. thanks very much for your help.
fred
test.tar.gz

[milot-mirdita]

I think you changed the output name of hhmake. Its building files like '1JSF.blits.hhm' now.
You can fix your workaround by doing something like this:

awk '{ gsub(/\.a3m/, ".blits.hhm"); print $0 }' sorting_a3m > sorting_hhm

However you should do this as the very first step after you build the a3m ffindex database, remove the file extensions from the index:

awk '{ gsub(/\.a3m/, ""); print $0 }' test_a3m.ffindex > test_a3m.ffindex_tmp && mv test_a3m.ffindex_tmp test_a3m.ffindex

I still have no idea what the root cause of your issue is, but please try to rebuild the hhm and cs219 ffindex database with this renamed test_a3m.ffindex. This should in general yield less issues (since all files will share the same key).

[fslee62]

hi milot,
yes that was it. it was my bad. i forgot the name prefixes matter. however, i still need to use 2 separate "sorting.dat" files - one for a3m and one for hhm. in the user guide, it seemed like the same "sorting.dat" can be used for both cases. how is that?
fred

[milot-mirdita]

I am not where your .hhm endings come from in your index files. ffindex_apply(_mpi) should continue to use the initial key (XXXX.a3m). So all the files SHOULD have the same keys, then sorting.dat should also work for all files.

[fslee62]

got it. perhaps a line in the user guide warning users about alternative names may be handy at times. thanks a whole bunch again. warm regards, fred