Module cache improvements

[clee-r7]

As part of the new data service changes (aka: Goliath) clean up of the current module caching was needed to allow an easier port to a data services model.

Changes:

1. Framework now ships with a baseline metadata module file under the "db" directory. While not necessary for startup it enhances the first time user experience by not needing all the modules to first be loaded into memory when running without a DB.
2. Whether running with or without a DB an on disk representation of the module metadata is now stored to disk: //store/. This file is generated from the base file mentioned above and may also include user specific modules.

Side effects:
When running without a DB:

1. Memory usage on startup was approximately 70-100mb less
2. Startup time was notably faster
3. Module search now in the ms range not the ~15+ sec range when tested

When running with a DB

1. Changes are negligible

[sempervictus]

Neat, thank you. Looks sane at first pass, will try to get this merged ob my end and test it out.

[sempervictus]

"#{type}/#{refname}" maybe?

[clee-r7]

thank you

[h00die]

metadata

[clee-r7]

thank you

[jmartin-tech]

The specs failing in ruby 2.3 look like they are things that should either be updated or removed.

Failed examples:

rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:1]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache should create cache entry for path
rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:2:3]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache cache entry
rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:2:4]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache cache entry
rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:2:1]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache cache entry
rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:2:2]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache cache entry
rspec './spec/lib/msf/core/module_manager_spec.rb[1:1:8:2:3:2:1]' # Msf::ModuleManager it should behave like Msf::ModuleManager::Cache #module_info_by_path_from_database! with database cache typed module set without reference_name should set reference_name value to Msf::SymbolicModule

[clee-r7]

My last commit has a one line that shouldn't have caused the tests to be failing. How do I kick off the tests again (without recommit)?

[busterb]

Click 'Details' on the Travis CI build failed link above. Make sure you login to Travis CI with your github credentials first, you'll have a 'Restart build' button available then.

[jmartin-tech]

I restarted test again, I expect may still fail.

For me the tests fail on 2.3.5 but not on 2.4.2 and I ran locally on Linux to validate.

[jmartin-tech]

I would suggest adding 'db/modules_metadata_base.pstore' to .gitignore. While it should be updated at times this would enforce explicit add of the file so user local data does not have end up getting added/leaked into master. We can add periodic update to the weekly release process to keep this from becoming a requirement to update each time a module changes or is added.

[clee-r7]

I kicked off the builds this morning and they are all passing now :/

[jmartin-tech]

@chlee: "I kicked off the builds this morning and they are all passing now :/"

That makes me sad, I validated externally for the failed tests, wonder why it would start passing in travis now :-(

Now looks like it is having a failure in 2.4.2, I suspect this is timing related, some tests are expecting caching to complete before run. We need to get consistent results before we land this.

[clee-r7]

@jmartin-r7 - I think I see the problem - despise timing tests

[busterb]

Landed, as the first Metasploit 5 PR. Thanks @clee-r7!

[busterb]

Added 86c927e to fix a problem with msfvenom running.

[busterb]

Release Notes

The module search system has been overhauled to require less memory, return results faster, and to not require a database to work at full speed. New module metadata and search options are also included in this back-ported feature from Metasploit 5.