Error in search_fulltext.py

[sebastian0x00]

Hi! have a problem running the command python3 search_fulltext.py -g -s >cve.json

He shows me the following

XXXXX@ubuntu:~/cve-search/bin$ python3 search_fulltext.py  -g -s >cve.json
Traceback (most recent call last):
  File "search_fulltext.py", line 28, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')

I have Whoosh==2.7.0

Is it a bug? Or am I doing something wrong?

Thank you

[adulau]

It seems that you have no indexes in the whoosh directory. It could be that the indexing was not done at the initial import. You can run manually the indexing by doing the following command:

python3 db_fulltext.py -l 0 -v 

and then do search again using search_fulltext.py. If you any issue, let me know.

[sebastian0x00]

@adulau Thanks for reply.
I executed the command python3 db_fulltext.py -l 0 -v and after a few minutes, ended correctly.

But running the command python3 search_fulltext.py -g -s >cve.json Same error:

XXXX@ubuntu:~/cve-search/bin$ python3 search_fulltext.py -g -s >cve.json
Traceback (most recent call last):
  File "search_fulltext.py", line 28, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')

Thanks

[sebastian0x00]

hi @adulau
Any news?
Thanks :D

[adulau]

It seems that the indexdir is not at the right place. Could you check where you Whoosh indexes were created? You can update it from the tool itself (I still need to update the code to use the directory path from the config file)

[sebastian0x00]

@adulau Thanks Alexandre.
But.. how check where my Whoosh indexdir were created?
I am very very newbie in linux.

I googled it (whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')) but i couldn't find any solution

@PidgeyL This error occurs in the same configuration bug that you helped me fix it.
You can check if you have my new problem ?

[sebastian0x00]

Hi @adulau @PidgeyL
Any News?
Thanks! :D

[sebastian0x00]

Hello @adulau @PidgeyL
Any News?
I tried searching on Google but not found anything.
Thanks!

[adulau]

Could you just check if you have the index built by db_fulltext.py? it's in indexdir with files likes MAIN_*.

[sebastian0x00]

@adulau Thanks for reply.
I executed the command python3 db_fulltext.py -l 0 -v and after a few minutes, ended correctly.

But running the command python3 search_fulltext.py -g -s >cve.json Same error :(

I have a folder called " idexdir " with these files

This is correct? Or wrong?

[lvets]

I didn't search for related issues when I opened #115, sorry for that.

Had some time to do some additional debugging today:
If I change indexpath = "./indexdir" to indexpath = "/home/me/Source/cve-search/indexdir/" (the directory of the index) in search_fulltext.py it seems to work:

me@zuuuul:~/Source/cve-search/bin$ python3 search_fulltext.py
usage: search_fulltext.py [-h] [-q Q] [-t] [-f] [-m M] [-l] [-g] [-s] [-n]
                          [-r]

Full text search for cve-search

optional arguments:
  -h, --help  show this help message and exit
  -q Q        query to lookup (one or more)
  -t          output title of the match CVE(s)
  -f          output matching CVE(s) in JSON
  -m M        most frequent terms in CVE description (m is top-m values)
  -l          dump all terms encountered in CVE description
  -g          graph of most frequent terms with each matching CVE (JSON
              output)
  -s          enable stemming on graph JSON output (default is False)
  -n          lookup complete cpe (Common Platform Enumeration) name for
              vulnerable configuration
  -r          lookup ranking of vulnerable configuration
me@zuuuul:~/Source/cve-search/bin$

I'll keep looking as well.

[sebastian0x00]

thanks @tunkaflux

I tried what you say and do not work for me :(

in the folder /home/XXX/cve-search/bin I modified the file search_fulltext.py for:

indexpath = "/home/XXX/cve-search/sbin/indexdir/"

Then, i run
/home/XXX/cve-search/sbin/python3 db_fulltext.py

And Same error.

XXX@XXX:~/cve-search/bin$ python3 search_fulltext.py  -g -s >cve.json
Traceback (most recent call last):
  File "search_fulltext.py", line 28, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
XXX@XXX:~/cve-search/bin$ 

this is strange , because in the folder indexdir files MAIN exist

XXX@XXX:~/cve-search/sbin/indexdir$ ls
_MAIN_182025.toc           MAIN_mrk9sxg7a3yov83a.seg  MAIN_t0njwzx0d0erltv3.seg
MAIN_2sqhgn9dgsziop36.seg  MAIN_n5k6rxn2bhlno8c3.seg  MAIN_txtxc3b7k3ehqa6f.seg
MAIN_87ifw0zj2zi56f13.seg  MAIN_ophh6mwh8wiiejlz.seg  MAIN_WRITELOCK
MAIN_gh9o0059dz16kmzh.seg  MAIN_pbf1ag3eusak09bb.seg  MAIN_zczkznex6ipiiwnt.seg
MAIN_i89a142xfe6be5vk.seg  MAIN_qpxm9zfq73e0xpbw.seg
MAIN_lz4f7r48zqz662p3.seg  MAIN_r7e3w0au0zfiia19.seg

[lvets]

Ok... but why did you close this? :)

[sebastian0x00]

@tunkaflux Sorry, my bad :P

[lvets]

Also, I noticed my indexdir is in "/home/xxx/cve-search/indexdir/". That's also the directory that gets updated when I reindex everything.
It seems you are using "/home/xxx/cve-search/sbin/indexdir"? Can you reindex everyhing and see which dir gets updated: cve-search/indexdir or cve-search/sbin/indexdir?

Afterwards, point to the updated indexdir and try again.

[lvets]

For the record, both indexdirs work for me, so are you sure that you didn't make a typo somewhere?

[sebastian0x00]

@tunkaflux
Thank you very much for answering
This is strange, i reindex everything and still creates the MAIN files in /sbin/indexdir but search_fulltext.py not working.
I re- download CVE-SERCH and I used the command ( who told me @adulau) python3 db_fulltext.py -l 0 -v to reindex and not working.
Also modified the file search_fulltext.py (change indexpath) and I still generates the same problem

Which command you used to reindex everything?
What operating system and version used? So I try the same specifications that you have

Thanks!

[PidgeyL]

I'm running Ubuntu 14.04, 64 and 32 bit

[lvets]

Ubuntu 14.04, 64 bit here.

[sebastian0x00]

Hi!
Not work for me :( Install a new Ubuntu (14.04) and install/extract CVE-SEARCH (new dowload) and the same error.

python3 search_fulltext.py

Traceback (most recent call last):
  File "search_fulltext.py", line 30, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')

BTW, now the MAIN files are in the folder "cve-search/indexdir"

any ideas? @adulau @tunkaflux @PidgeyL

If you want, I can upload my image VirtualBox Ubuntu so they can check or revise my problem.

[sebastian0x00]

Hi people!
any news or suggestions?

[lvets]

@sugusrojo Have you tried fix #97 from the Adulau repo? See commit (adulau@98bda7d)

[sebastian0x00]

@tunkaflux tThanks for the Reply!
Yes, I've tried.
I tried new fresh installation Ubuntu 14.04 and re dowloding a new copy CVE-SEARCH.
now, the MAIN files are in the folder "cve-search/indexdir" but the same error occurs.

[sebastian0x00]

@tunkaflux you can upload your Ubuntu Virtual Image?
Because...
You CVE-SEARCH in Ubuntu works.
My CVE-SEARCH not work in Ubuntu .
I want to see the difference between these two OS configuration or see what's difference.

[lvets]

@sugusrojo No, unfortunately, I can't upload an image.

@adulau @PidgeyL: Does search work for you guys?

[ghost]

I've tried it as well and got the same kind of error. I let the VM run at the office as it's currently generating this index. I'll retry tomorrow morning when it's done.

PS: Running this on Ubuntu Server 14.04-3 LTS x86_64.

[sebastian0x00]

Thanks @AkiraTakizawa . I'll wait for your answer.

[adulau]

The last commit should be fine.

Here is an example of the use of the full-text indexing:

[adulau:~/git/cve-search]↥ master* ± python3 ./sbin/db_fulltext.py -l 10 -v 
Indexing CVE-ID CVE-2015-8581 The EjbObjectInputStream class in Apache TomEE allows remote attackers
Indexing CVE-ID CVE-2015-8580 Multiple use-after-free vulnerabilities in the (1) Print method and (2
Indexing CVE-ID CVE-2015-8566 The Session package 1.x before 1.3.1 for Joomla! Framework allows remo
Indexing CVE-ID CVE-2015-8565 Directory traversal vulnerability in Joomla! 3.2.0 through 3.3.x and 3
Indexing CVE-ID CVE-2015-8564 Directory traversal vulnerability in Joomla! 3.4.x before 3.4.6 allows
Indexing CVE-ID CVE-2015-8563 Cross-site request forgery (CSRF) vulnerability in the com_templates c
Indexing CVE-ID CVE-2015-8562 Joomla! 1.5.x, 2.x, and 3.x before 3.4.6 allow remote attackers to con
Indexing CVE-ID CVE-2015-8476 Multiple CRLF injection vulnerabilities in PHPMailer before 5.2.14 all
Indexing CVE-ID CVE-2015-8370 Multiple integer underflows in Grub2 1.98 through 2.02 allow physicall
Indexing CVE-ID CVE-2015-8358 Directory traversal vulnerability in the bitrix.mpbuilder module befor
Processing[##################################################] 10/10
[adulau:~/git/cve-search]↥ master* ± python3 ./bin/search_fulltext.py -q TomEE
CVE-2015-8581
[adulau:~/git/cve-search]↥ master* ± 

I hope this helps.

[ghost]

Good morning
I confirm that it works after (re)building the index :)

Example:

sadao@kujira:~/cve-search$ python3 ./bin/search_fulltext.py -q Lync
CVE-2013-1302
CVE-2015-2431
CVE-2014-4068
CVE-2015-2510
CVE-2015-6061
CVE-2014-4071
CVE-2015-1671
CVE-2015-2532
CVE-2015-2435
CVE-2014-4070
CVE-2015-2536
CVE-2014-1823
CVE-2012-1849
CVE-2015-2464
CVE-2015-2463
CVE-2015-2531
CVE-2015-2456
CVE-2015-2455
CVE-2014-1818
CVE-2014-1817
CVE-2015-6106
CVE-2015-6107
CVE-2015-6108
CVE-2012-1858
CVE-2013-3906
CVE-2012-2520
CVE-2013-3129
CVE-2015-2503

[adulau]

@AkiraTakizawa Thank you for the test.

[sebastian0x00]

I found the problem! @adulau @AkiraTakizawa @tunkaflux @PidgeyL

If I run the command from the folder CVE-search WORKS

XXX@XXX:~/cve-search$ python3 ./bin/search_fulltext.py -q CVE-2013-1455

If I run it from the CVE-search\bin folder does NOT WORK! It fails.

XX@CXXX:~/cve-search/bin$ python3 search_fulltext.py -q CVE-2013-1455
Traceback (most recent call last):
  File "search_fulltext.py", line 30, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')

Example:

[PidgeyL]

Thanks for your help! That probably means there is still a relative path that is not working yet. I'll take this issue and work on it on my way home from work, and if fixed, I'll commit it tonight

[PidgeyL]

I am currently testing this, but it works perfectly for me in both cases. Can you do the following for me?

1. give me the value for Indexdir in el./etc/configuration.ini
2. add print(indexpath) to line 27 of ./bin/search_fulltext.py and give the output?

[sebastian0x00]

Thanks @PidgeyL

1. I haven't file configuration.ini in cve-search/etc.
   Only "configuration.ini.sample"

[Redis]
Host: localhost
Port: 6379
VendorsDB: 10
NotificationsDB: 11
RefDB: 12
[Mongo]
Host: localhost
Port: 27017
DB: cvedb
[dbmgt]
Tmpdir: ./tmp/
[FulltextIndex]
Indexdir: ./indexdir/
[Sources]
CVE: http://static.nvd.nist.gov/feeds/xml/cve/
CPE: http://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.2.xml
CWE: http://cwe.mitre.org/data/xml/cwec_v2.8.xml.zip
d2sec: http://www.d2sec.com/exploits/elliot.xml
vFeed: http://www.toolswatch.org/vfeed/vfeed.db.tgz
vFeedStatus: http://www.toolswatch.org/update.dat
Vendor: https://nvd.nist.gov/download/vendorstatements.xml
CAPEC: http://capec.mitre.org/data/xml/capec_v2.6.xml
MSBULLETIN: http://download.microsoft.com/download/6/7/3/673E4349-1CA5-40B9-8879-095C72D5B49D/BulletinSearch.xlsx
Ref: https://cve.mitre.org/data/refs/refmap/allrefmaps.zip
exploitdb: https://github.com/offensive-security/exploit-database/raw/master/files.csv
[Webserver]
Host: 127.0.0.1
Port: 5000
Debug: True
PageLength: 50
LoginRequired: False
SSL: True
Certificate: ssl/cve-search.crt
Key: ssl/cve-search.key
[Logging]
Logging: True
Logfile: log/cve-search.log
MaxSize: 150MB
Backlog: 5
[CVE]
DefaultCVSS: 5
StartYear: 2002

Add line 27

/home/XXX/cve-search/lib/.././indexdir
Traceback (most recent call last):
  File "search_fulltext.py", line 30, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')

[PidgeyL]

And if you do step two from the above directory?

[sebastian0x00]

It's strange @PidgeyL
Both in the same location!
But fails from CVE-search \ bin

Thanks

csirt@CSIRTCVE:~/cve-search$ python3 ./bin/search_fulltext.py -q CVE-2013-145
/home/csirt/cve-search/lib/.././indexdir
csirt@CSIRTCVE:~/cve-search/bin$

csirt@CSIRTCVE:~/cve-search$ cd bin
csirt@CSIRTCVE:~/cve-search/bin$ python3 search_fulltext.py -q CVE-2013-145
/home/csirt/cve-search/lib/.././indexdir
Traceback (most recent call last):
  File "search_fulltext.py", line 30, in <module>
    ix = index.open_dir("indexdir")
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
    % (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
csirt@CSIRTCVE:~/cve-search/bin$ 

[PidgeyL]

I'm baffled... Could you gzip your entire folder and send it to me, either by mail or upload it somewhere? That would be a lot easier to debug