Efilter engine import failure during initialize #275

mutedmouse · 2015-10-14T20:52:26Z

Below is a trimmed copy of the error encountered during grr_config_updater initialize portion. Confirmed efilter was latest version with pip.
File "/usr/local/lib/python2.7/dist-packages/rekall/entities/init.py", line 3, in
from rekall.entities import query
File "/usr/local/lib/python2.7/dist-packages/rekall/entities/query/init.py", line 3, in
from rekall.entities.query import analyzer
File "/usr/local/lib/python2.7/dist-packages/rekall/entities/query/analyzer.py", line 29, in
from efilter import engine
ImportError: cannot import name engine

FAILURE RUNNING: grr_config_updater initialize

Ty.

destijl · 2015-10-14T21:22:06Z

It seems efilter has changed its structure enough that the latest version doesn't work with the version of rekall that we have.

Try installing the old version

 pip install efilter==1438631774

Pinning this version in our requirements.txt is hopefully enough to fix this properly.

$ virtualenv ~/TEMP
Installing setuptools, pip...done.
$ source ~/TEMP/bin/activate
(TEMP)$ pip install efilter==1438631774
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting efilter==1438631774
  Using cached efilter-1438631774.tar.gz
Collecting python-dateutil>2 (from efilter==1438631774)
  Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting pytz>=2011k (from efilter==1438631774)
  Using cached pytz-2015.6-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>2->efilter==1438631774)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, pytz, python-dateutil, efilter
  Running setup.py install for efilter
Successfully installed efilter-1438631774 python-dateutil-2.4.2 pytz-2015.6 six-1.10.0
(TEMP)$ python -c "from efilter import engine"

*works*

(TEMP)$ pip install --upgrade efilter
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting efilter from https://pypi.python.org/packages/source/e/efilter/efilter-1440489265.tar.gz#md5=f8aa013eeb3a262b8d74ac2b40b3709b
  Downloading efilter-1440489265.tar.gz
Requirement already up-to-date: python-dateutil>2 in ./TEMP/lib/python2.7/site-packages (from efilter)
Requirement already up-to-date: pytz>=2011k in ./TEMP/lib/python2.7/site-packages (from efilter)
Requirement already up-to-date: six>=1.5 in ./TEMP/lib/python2.7/site-packages (from python-dateutil>2->efilter)
Installing collected packages: efilter
  Found existing installation: efilter 1438631774
    Uninstalling efilter-1438631774:
      Successfully uninstalled efilter-1438631774
  Running setup.py install for efilter
Successfully installed efilter-1440489265
(TEMP)$ python -c "from efilter import engine"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name engine

destijl · 2015-10-14T21:34:51Z

Was this an install on a new clean system? Was it ubuntu trusty? I haven't been able to reproduce this yet, so please let me know if the above fixes it.

mutedmouse · 2015-10-14T21:39:31Z

Was on clean ubuntu server 14.04. It did not rectify the issue so far, I'm
backing each of the requirements off and trying the efilter install. I'll
post soon if it works.
On Oct 14, 2015 5:34 PM, "Greg" notifications@github.com wrote:

Was this an install on a new clean system? Was it ubuntu trusty? I haven't
been able to reproduce this yet, so please let me know if the above fixes
it.

—
Reply to this email directly or view it on GitHub
#275 (comment).

grrrrrrrrr · 2015-10-14T21:43:29Z

Can you go and delete all your Rekall related .pyc's? I remember some
issues with this name, I think it was a file first and is now a directory
but removing the .py could still keep an old .pyc around and then it would
pick that one up before the directory with the same name.

Cheers,
-Andy

On Wed, Oct 14, 2015 at 11:39 PM Muted Mouse notifications@github.com
wrote:

Was on clean ubuntu server 14.04. It did not rectify the issue so far, I'm
backing each of the requirements off and trying the efilter install. I'll
post soon if it works.
On Oct 14, 2015 5:34 PM, "Greg" notifications@github.com wrote:

Was this an install on a new clean system? Was it ubuntu trusty? I
haven't
been able to reproduce this yet, so please let me know if the above fixes
it.

—
Reply to this email directly or view it on GitHub
#275 (comment).

—
Reply to this email directly or view it on GitHub
#275 (comment).

destijl · 2015-10-14T21:44:20Z

@the80srobot for efilter

mutedmouse · 2015-10-14T21:54:54Z

Deleting all rekal py worked.

for path in locate rekal; do rm -rf $path; done

After that install_script_ubuntu.sh worked like a charm.

Thanks guys.
Can you go and delete all your Rekall related .pyc's? I remember some
issues with this name, I think it was a file first and is now a directory
but removing the .py could still keep an old .pyc around and then it would
pick that one up before the directory with the same name.

Cheers,
-Andy

On Wed, Oct 14, 2015 at 11:39 PM Muted Mouse notifications@github.com
wrote:

Was on clean ubuntu server 14.04. It did not rectify the issue so far, I'm
backing each of the requirements off and trying the efilter install. I'll
post soon if it works.
On Oct 14, 2015 5:34 PM, "Greg" notifications@github.com wrote:

Was this an install on a new clean system? Was it ubuntu trusty? I
haven't
been able to reproduce this yet, so please let me know if the above
fixes
it.

—
Reply to this email directly or view it on GitHub
#275 (comment).

—
Reply to this email directly or view it on GitHub
#275 (comment).

—
Reply to this email directly or view it on GitHub
#275 (comment).

pidydx · 2015-10-14T22:10:10Z

I just started having this problem too when switching over to the install from source script. I had to blow away all of rekall and reinstall it to get past it. I am doing some more testing now.

mutedmouse · 2015-10-14T22:21:01Z

I ended up blowing away rekal and installing with:
pip uninstall efilter python-dateutil pytz six
for path inlocate rekal; do rm -rf $path; done
pip install efilter==1438631774

Then run the install script:
./install_script_ubuntu.sh

Those actions fixed it for me, now operational.
On Oct 14, 2015 6:10 PM, "Sean Gillespie" notifications@github.com wrote:

I just started having this problem too when switching over to the install
from source script. I had to blow away all of rekall and reinstall it to
get past it. I am doing some more testing now.

—
Reply to this email directly or view it on GitHub
#275 (comment).

destijl · 2015-10-14T22:26:05Z

Awesome, I've changed the requirements.txt to pin efilter and will push it shortly.

#275

pidydx · 2015-10-14T22:40:39Z

I removed rekall and rekall-core. Reinstalled deps via the ubuntu script and everything was ok. I upgraded to latest efilter and got the 'expressions' import error. I removed rekall-core and installed rekall-core and it force downgraded efilter.

Removing and reinstalling rekall-core appears to be the fix. I haven't fully tested a from scratch clean install, but unless there is an install order problem it should likely not be impacted.

destijl · 2015-10-14T22:48:18Z

I'm running in a clean VM now to verify before I close this out.

pidydx · 2015-10-14T22:49:03Z

Just finished my clean install test and it worked.

destijl · 2015-10-14T23:19:54Z

Works for me, thanks for the reports @pidydx and @mutedmouse

mutedmouse · 2015-10-15T00:26:05Z

Thanks everyone.
On Oct 14, 2015 6:49 PM, "Sean Gillespie" notifications@github.com wrote:

Just finished my clean install test and it worked.

—
Reply to this email directly or view it on GitHub
#275 (comment).

scudette · 2015-10-15T04:33:06Z

You should not install efilter itself since head is broken. When you install rekall it pulls in the right version that it works with:

pip install rekall

works. Do not however try to upgrade efilter:

pip install efilter

since this will will in the latests efilter version and break rekall.

the80srobot · 2015-10-15T08:27:57Z

Hi,

So the install from source script for some reason ignores the version in
setup.py - I am not sure why it does that, but for now, if you install
rekall from pip (not setup.py install) it will pull in the right version of
efilter.

We will be updating Rekall soon to work with latest efilter and then this
will no longer be a problem. Sorry about the inconvenience.

Adam

On Thu, Oct 15, 2015 at 6:33 AM Michael Cohen notifications@github.com
wrote:

You should not install efilter itself since head is broken. When you
install rekall it pulls in the right version that it works with:

pip install rekall

works. Do not however try to upgrade efilter:

pip install efilter

since this will will in the latests efilter version and break rekall.

—
Reply to this email directly or view it on GitHub
#275 (comment).

destijl · 2015-10-15T16:10:37Z

I confirmed the rekall efilter pinning works as below, so I'm still not sure how this failed. I couldn't reproduce the original problem with the GRR install, but there were multiple reports. We are using pip and requirements.txt to install.

@mutedmouse you were using the latest install script from the repo, right? If you were using an old one that might explain it...

$ source TEMP/bin/activate
(TEMP)$ pip install rekall-core==1.4.1
[snip]
Collecting efilter==1438631774 (from rekall-core==1.4.1)
  Using cached efilter-1438631774.tar.gz
[snip]
Successfully installed PyAFF4-0.17 PyYAML-3.11 SPARQLWrapper-1.6.4 acora-1.9 distorm3-3.3.0 efilter-1438631774 html5lib-0.9999999 intervaltree-2.1.0 isodate-0.5.4 pycrypto-2.6.1 pyelftools-0.23 pyparsing-2.0.3 python-dateutil-2.4.2 pytz-2015.6 rdflib-4.2.1 rekall-core-1.4.1 six-1.10.0 sortedcontainers-0.9.6
(TEMP)$ python -c "from efilter import engine"

destijl · 2015-10-15T17:24:01Z

I installed the full requirements.txt in a clean virtualenv and got the same (correct) result, so I'm still not clear on how this ever failed, but I'll keep the explicit efilter install in for now.

mutedmouse · 2015-10-15T21:19:30Z

I had pulled the install yesterday afternoon from the github grr master
branch.
On Oct 15, 2015 1:24 PM, "Greg" notifications@github.com wrote:

I installed the full requirements.txt in a clean virtualenv and got the
same (correct) result, so I'm still not clear on how this ever failed, but
I'll keep the explicit efilter install in for now.

—
Reply to this email directly or view it on GitHub
#275 (comment).

the80srobot · 2015-10-17T20:41:00Z

Greg, I can reproduce pip ignoring the version requirements with the
darwinbuild.sh script (from rekall's repo). As far as I can tell it almost
looks like a bug in pip.

On Thu, Oct 15, 2015 at 11:19 PM Muted Mouse notifications@github.com
wrote:

I had pulled the install yesterday afternoon from the github grr master
branch.
On Oct 15, 2015 1:24 PM, "Greg" notifications@github.com wrote:

I installed the full requirements.txt in a clean virtualenv and got the
same (correct) result, so I'm still not clear on how this ever failed,
but
I'll keep the explicit efilter install in for now.

—
Reply to this email directly or view it on GitHub
#275 (comment).

—
Reply to this email directly or view it on GitHub
#275 (comment).

cvandeplas · 2015-11-17T11:18:21Z

I had this issue on a SIFT system.
Rekall 1.2.0 and rekall-core-1.4 seemed to be installed (after the error occured).
Doing a sudo pip uninstall rekall rekall-core and then sudo pip install rekall rekall-core rekall-gui updated rekall to v1.4 and solved the issue

kftp · 2015-12-23T13:28:04Z

Hi,
Having faced this efilter issue, I gave a try to the ideas suggested above in this post (including removal of all rekall references) ; however, I now have a new error ImportError: No module named rekall when running grr_config_updater initialize.
=> I then did pip install rekall rekall-core rekall-gui install => still no success
Thanks in advance for you help ;)

scudette · 2015-12-23T13:37:46Z

you need to remove all efilter references - somehow you have a new version
of efilter installed on your system which is not compatible with the old
rekall version.

Alternatively just update rekall to the latest version - git clone it then
cd rekall-core/ then python setup.py install (do not install from setup.py
in the top level directory since it will just pull in the pip distribution

it is a metapackage).

On 23 December 2015 at 14:28, kftp notifications@github.com wrote:

Hi,
Having faced this efilter issue, I gave a try to the ideas suggested above
in this post (including removal of all rekall references) ; however, I now
have a new error ImportError: No module named rekall when running grr_config_updater
initialize.
=> I then did pip install rekall rekall-core rekall-gui install =>
still no success
Thanks in advance for you help ;)

—
Reply to this email directly or view it on GitHub
#275 (comment).

kftp · 2015-12-23T14:41:46Z

Thanks for getting back to me so quickly. I just ran the setup.py install (the one in rekall-core) ; it resulted in a memory error :(
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
MemoryError

My SIFT VM is set at 2GB RAM. Is this not enough ? I read here (https://github.com/google/grr-doc/blob/master/admin.adoc#memory-limit) that GRR typical memory consumption is 30GB, however I do not have a server to just test GRR. And so far it is about installing, not yet using :-D

Full end of setup.py execution trace is

GEN capstone.pc
Traceback (most recent call last):
File "setup.py", line 148, in
install_requires=install_requires,
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/install.py", line 67, in run
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/install.py", line 117, in do_egg_install
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 380, in run

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 610, in easy_install

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 661, in install_item

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 709, in process_distribution

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/pkg_resources/init.py", line 824, in resolve
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/pkg_resources/init.py", line 1069, in best_match
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/pkg_resources/init.py", line 1081, in obtain
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 629, in easy_install

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 659, in install_item

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 842, in install_eggs

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 1070, in build_and_install

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/command/easy_install.py", line 1056, in run_setup

File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 240, in run_setup
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 193, in setup_context
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 152, in save_modules
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 126, in exit
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
File "/usr/local/lib/python2.7/dist-packages/setuptools-15.0-py2.7.egg/setuptools/sandbox.py", line 110, in dump
MemoryError*

destijl added a commit that referenced this issue Oct 14, 2015

Fix efilter dependency bug

99606f2

#275

destijl closed this as completed Oct 14, 2015

grrrrrrrrr mentioned this issue Dec 23, 2015

install fails / efilter #318

Closed

snyk-bot mentioned this issue Jan 25, 2022

[Snyk] Security upgrade grunt-contrib-jshint from 0.11.3 to 3.1.0 ekmixon/grr#33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Efilter engine import failure during initialize #275

Efilter engine import failure during initialize #275

mutedmouse commented Oct 14, 2015

destijl commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

grrrrrrrrr commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

pidydx commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

destijl commented Oct 14, 2015

pidydx commented Oct 14, 2015

destijl commented Oct 14, 2015

pidydx commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 15, 2015

scudette commented Oct 15, 2015

the80srobot commented Oct 15, 2015

destijl commented Oct 15, 2015

destijl commented Oct 15, 2015

mutedmouse commented Oct 15, 2015

the80srobot commented Oct 17, 2015

cvandeplas commented Nov 17, 2015

kftp commented Dec 23, 2015

scudette commented Dec 23, 2015

kftp commented Dec 23, 2015

Efilter engine import failure during initialize #275

Efilter engine import failure during initialize #275

Comments

mutedmouse commented Oct 14, 2015

destijl commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

grrrrrrrrr commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

pidydx commented Oct 14, 2015

mutedmouse commented Oct 14, 2015

destijl commented Oct 14, 2015

pidydx commented Oct 14, 2015

destijl commented Oct 14, 2015

pidydx commented Oct 14, 2015

destijl commented Oct 14, 2015

mutedmouse commented Oct 15, 2015

scudette commented Oct 15, 2015

the80srobot commented Oct 15, 2015

destijl commented Oct 15, 2015

destijl commented Oct 15, 2015

mutedmouse commented Oct 15, 2015

the80srobot commented Oct 17, 2015

cvandeplas commented Nov 17, 2015

kftp commented Dec 23, 2015

scudette commented Dec 23, 2015

kftp commented Dec 23, 2015