Python silent sqlite compile error #15300

Closed
staticfloat opened this Issue Oct 4, 2012 · 68 comments

Projects

None yet
Contributor

Symptom: python -c import sqlite fails:

$ python -c 'import sqlite'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named sqlite

Cause: Python build silently passes over a _sqlite module build error. It's a large log that @samueljohn requested, the important bits are around line 918 in python-install.log. sqlite-install.log is at the bottom.

I checked to make sure that Python is not linking against system python:

$ otool -L `which python`
/Users/sabae/.homebrew/bin/python:
        /Users/sabae/.homebrew/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Any ideas?

Contributor

Thanks, I'll have a look. But I could not reproduce on 10.8. Perhaps the log gives a clue.

Contributor

And with --env=std is builds?
From the log, I don't really get any better clue.

At least I know that I have seen this error, if Apple's sqlite is used and that does not have loadable-extensions enabled. Worth a try: brew rm sqlite and brew install sqlite ?

Contributor

I already redid sqlite, and posted my install log below the python install log, no difference.

And yes, confirmed that brewing with --std=env builds sqlite just fine.

Contributor

Hmm. What is your env?
I have just tried to install homebrew into my home under .homebrew. A plain new homebrew and brew install python -v sadly just works. Or maybe not so sadly.

But still I don't have a clue, why it fails for you.

Contributor

From my build log (I know this is from the build of the tkinter module, but python spits out the details only for unsuccessful modules):

cc 
-fno-strict-aliasing -fno-common -dynamic 
-I/Users/sam/.homebrew/include 
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk 
-DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DWITH_APPINIT=1 
[I removed tk specific headers here.]
-I/private/tmp/python-SdCZ/Python-2.7.3/Mac/Include 
-I. -IInclude -I./Include 
-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include 
-I/usr/local/include 
-I/private/tmp/python-SdCZ/Python-2.7.3/Include 
-I/private/tmp/python-SdCZ/Python-2.7.3 
-c /private/tmp/python-SdCZ/Python-2.7.3/Modules/_tkinter.c 
-o build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_tkinter.o 
-framework Tk

brew: superenv removed: 
  -I/Users/sam/.homebrew/include 
  -g -O3 -Wall -Wstrict-prototypes 
  -I./Include 
  -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include 
  -I/private/tmp/python-SdCZ/Python-2.7.3/Include 
  -I/private/tmp/python-SdCZ/Python-2.7.3

From your log:

cc 
-fno-strict-aliasing -fno-common -dynamic 
-I/Users/sabae/.homebrew/include 
-DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DMODULE_NAME="sqlite3" 
-IModules/_sqlite 
-I/usr/include 
-I/private/tmp/python-2PJR/Python-2.7.3/Mac/Include 
-I. -IInclude -I./Include 
-I/private/tmp/python-2PJR/Python-2.7.3/Include -I/private/tmp/python-2PJR/Python-2.7.3 
-c /private/tmp/python-2PJR/Python-2.7.3/Modules/_sqlite/util.c -o build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-2PJR/Python-2.7.3/Modules/_sqlite/util.o

brew: superenv removed: 
  -I/Users/sabae/.homebrew/include 
  -g -O3 -Wall -Wstrict-prototypes 
  -I./Include 
  -I/private/tmp/python-2PJR/Python-2.7.3/Include 
  -I/private/tmp/python-2PJR/Python-2.7.3

(I added some newlines)

Abstracting over the differences between tk and sqlite, I don't see a real problem here.

At least, I think it's not a problem, what superenv removes here, but what it adds (and that is not shown in the log).
Remember what we see here is what Python passes to the cc. Homebrew's superenv injects additional paths that are not displayed here. (because the cc is a wrapper script created by homebrew).

To investigate further, I need the ~/Library/Los/Homebrew/cc.log which is created when you export HOMEBREW_LOG=1 and then brew install python -v.
(Someone has to update man brew to include this, I think)


Here is the interesting part of my cc.log. It's the last block that contains the string "sqlite". I added newlines.:

[/Users/sam/.homebrew/Library/ENV/4.3/cc -Osa] xcrun clang 
-pipe -w -Os -march=native -bundle -undefined dynamic_lookup 
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/cache.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/connection.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/cursor.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/microprotocols.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/module.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/prepare_protocol.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/row.o build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/statement.o 
build/temp.macosx-10.8-x86_64-2.7/private/tmp/python-SdCZ/Python-2.7.3/Modules/_sqlite/util.o 
-L/usr/local/lib 
-lsqlite3 
-o build/lib.macosx-10.8-x86_64-2.7/_sqlite3.so 
-Wl,-search_paths_first
--sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk 
-isystem/Users/sam/.homebrew/include -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include 
-isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/libxml2 
-isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/apache2 
-isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/System/Library/Frameworks/Python.framework/Versions/Current/include/python2.7 
-isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers/ 
-I/Users/sam/.homebrew/opt/readline/include -L/Users/sam/.homebrew/opt/readline/lib 
-L/Users/sam/.homebrew/lib 
-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/lib 
-L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries

I have a Xcode-only setup, but my CLT-only system builds python with sqlite just fine, too. However, I should test the CLT system with homebrew in a non-standard dir.

The only thing which makes me itch when I look at my cc.log are the paths to -L/usr/local/lib. But apparently this doesn't lead to wrong linking. The sqlite module gets still linked to /Users/sam/.homebrew/lib/libsqlite3.0.8.6.dylib even if I have my production system in /usr/local/lib, including a sqlite.
I had expected to see the /Users/sam/.homebrew/opt/sqlite/lib to be in the -L at some point.

I think if we can debug this, we may be able to improve superenv.

Contributor

What I'd like to ask @mxcl at this point: Why do I find opt/readline/lib" but not opt/sqlite/lib in the -L flags in the cc.log? Both are dep of python (but readline is keg_only). Please teach me. I still suck so much at superenv.

Contributor

Here is the log, and I guess my only question is why is there a -L/usr/lib in there? That seems bad to me. :P

Member
mxcl commented Oct 5, 2012

What I'd like to ask @mxcl at this point: Why do I find opt/readline/lib" but not opt/sqlite/lib in the -L flags in the cc.log? Both are dep of python (but readline is keg_only). Please teach me. I still suck so much at superenv.

Because readline is keg-only and sqlite is not. stdenv has identical behavior.

And yes the bug would be that -L/usr/lib comes before -L/usr/local/lib.

The reason this doesn't effect stdenv is we put HOMEBREW's libpath first. superenv should be removing that line, but for some reason doesn't. So that's the bug, the library path fixing code has a bug.

I repeat again: superenv is simple, you are trying to think of it as a complicated system, all the problems we have had with it are bugs in my code.

Member
mxcl commented Oct 5, 2012

For me: we also need to remove: -Wl,-search_paths_first. Superenv chooses the search path order, not the build-tool.

Contributor

@mxcl yep, that answers my questions. So I leave the python formula as is.
Can't say much about the "search_paths_frist" thing.

Superenv is good. Don't get me wrong. I misunderstood the keg_only handling here.

If I install homebrew somewhere different than /usr/local the -L/usr/local/lib should also be removed. I don't know how it got there. It's not added by the formula. Perhaps Python's build system added it because we see it in both of our cc.log files even if we both don't have homebrew at /usr/local.
(Well my production system is there, but not for this test)

Alives commented Nov 5, 2012

How do I fix my path issue if I am using /usr/local/homebrew/bin as my directory?

Contributor

@Alives for you the SQLite module is broken too? It would help me diagnose I you could gist.github.com the output of your build brew install python --verbose. (You might need to brew rm python first.)
Also please post your brew --config and brew doctor.
Note to myself: I have to debug superenv to find where the -L/usr/lib slipped in.

Alives commented Nov 5, 2012

https://gist.github.com/4020694

FYI: I had to rm python first and that may have left some libs behind.. not sure.

Alives commented Nov 7, 2012

Should I not just edit the python formula for now to make a one-off fix for the time being?

Contributor

@Alives thanks, on your system, too, python attempts to link against the sqlite from OS X. I don't know a one-liner that fixes this. (Except disabling the dynamic loadable sqlite extensions)

Alives commented Nov 7, 2012

Can you point out the lines that show that? Maybe I can put a bandaid on it for the time being.

Contributor

@Alives, I'd rather try to find where and why -L/usr/lib slipped in :-)
Here is the line: https://github.com/mxcl/homebrew/blob/master/Library/Formula/python.rb#L95
Comment it out and python uses sqlite from OS X. That works, too but you won't have dynamically loadable extensions (which are rarely used).

Contributor
Sharpie commented Nov 11, 2012

Comment it out and python uses sqlite from OS X. That works, too but you won't have dynamically loadable extensions (which are rarely used).

Unless you're doing GIS work and need to load up SpatiaLite. I'll bet the QGIS formula in homebrew-science could be affected by this since it contains a large amount of algorithms written in Python.

A possible workaround could be to explicitly add opt/sqlite/lib to the linker path as is done with SpatiaLite to ensure it links against a SQLite lib that supports extension loading.

Ultimately, the solution may be to flag SQLite as keg_only so that LDFLAGS are automatically adjusted for any software that depends on it. However, I get the feeling from #15063 that superenv is not correctly adjusting LDFLAGS for keg_only software at the moment.

Contributor

sqlite: Yep, it is a dupe, therefore it should be keg_only. Always.

Besides that, I still have to debug the superenv + python behavior here. Why are the -L/usr/lib getting in there?

Contributor
adamv commented Jan 5, 2013

So...should we flag sqlite as keg-only? I forget what breaks when we do this.

Owner

Yes.

Contributor

brew uses sqlite

csync
gdal
konoha
libsoup
libspatialite
libzdb
pdns
python
python25
python26
python3
python31
python32
subversion
yarp
Contributor

@staticfloat do you still have the problem that sqlite is not build for python? I think this is related to homebrew not being in /usr/local.

Python seems to look there.
I have a test brew in /homebrew and I get:

xcrun otool -L /homebrew/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_sqlite3.so
/homebrew/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_sqlite3.so:
    /usr/local/lib/libsqlite3.0.8.6.dylib (compatibility version 9.0.0, current version 9.6.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

As we can see, it linkts to my "production" homebrew in /usr/local. This is a bug.

Contributor

@staticfloat since your test systems seems a good candidate:

Are you willing to help by temporary making
brew rm sqlite
brew rm python
brew edit sqlite and add keg_only to the formula.
And then brew install sqlite python?

Contributor

I tried to do this locally and indeed Python links to the correct sqlite if the sqlite formula would be keg_only.

Further, I tested csync and gdal and making sqlite keg_only works like charm.
xcrun otool -L $(which csync) shows it links the correct sqlite then.

Shall I open a PR for keg_only sqlite?

Contributor

this issue should be fixed now.

Contributor

I saw that your keg_only made it into sqlite, so I just updating and reinstalled sqlite and python, but I get the same error. Here's the full output of brew install -v -python. Note that it seems to try to fail on the _sqlite3.so module twice, once here and once here.

I double-checked, and sqlite is indeed keg_only right now.

Contributor

Have you set an ENV var to build universal?

Veerery strange.

Hi buddy i am getting the same Error and i have checked key_only is already there

Contributor

@farhatabbas please gist.github.com your output of brew doctor, brew --config, brew ls and their verbose build of python: brew install --verbose python and link here.

Perhaps I can see a pattern, then.
There must be some -L path that is added by python which should be filtered out by superenv.

clsung commented Jan 12, 2013

try
brew install sqlite3
brew link sqlite3
brew install python

Contributor

We just made SQLite keg_only, so brew link is intentionally skipped. But
when building python it will be added to the -L dirs.
however the problem some have here seems different: python passes a path to
the system SQLite which does not provide extensions and therefore the _sqlite module fails.

So I bet brew link won't help here much because that is how it was since
yesterday.

clsung commented Jan 13, 2013

I'm not sure. I encountered the same problem yesterday, and fix via
brew rm sqlite3; brew rm python
brew install sqlite3; brew link sqlite3
brew install python

So it might be worth a try if user who get in trouble with this :)

Contributor

The above does not work for me. I'm assuming this is because python looks for sqlite in /usr/local/ only.

jwsy commented Jan 14, 2013

Thanks, @clsung and @samueljohn - worked for me!
Don't forget to brew link --overwrite python to get python links restored.

Contributor

Again, brew link sqlite should not be required.

@samueljohn, unfortunately, I only got things working correctly with brew link sqlite before installing python. If I can help debug, let me know.

Contributor

@samueljohn; I fixed mine by changing the lines in distutils_fix_superenv() from:

# To allow certain Python bindings to find brewed software:
cflags = "CFLAGS=-I#{HOMEBREW_PREFIX}/include"
ldflags = "LDFLAGS=-L#{HOMEBREW_PREFIX}/lib"

to:

# To allow certain Python bindings to find brewed software:
cflags = "CFLAGS=-I#{HOMEBREW_PREFIX}/include -I#{Formula.factory('sqlite').include}"
ldflags = "LDFLAGS=-L#{HOMEBREW_PREFIX}/lib -L#{Formula.factory('sqlite').lib}"

I believe this is because, now that sqlite is keg_only, it must be manually added into the cflags and ldflags variables.

Contributor

@staticfloat is right that the part of setup.py that looks for sqlite include files only checks /usr/local/include among a couple others possibilities, which break sqlite3 support now that sqlite3 is keg-only. Adding #{Formula.factory('sqlite').include} to CFLAGS and LDFLAGS worked great for me!

Contributor

@staticfloat I am going to incorporate your suggestion into the python formula hoping we can finally get rid of the issues discussed here. Looks reasonable to me. Note: In order to NOT break if we brew upgraade sqlite, we should use the "#{Formula.factory('sqlite').opt_prefix}/include" (and lib) instead directly using the .include in the prefix.

+1 for this issue. Performed a fresh install of Homebrew today to see if samueljohn's fixes worked to no avail. Still getting the following when launching Sickbeard.

Traceback (most recent call last):
  File "sickbeard.py", line 49, in <module>
    import sickbeard
  File "/Users/Peter/Applications/SB/Master/sickbeard/__init__.py", line 23, in <module>
    import sqlite3
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

Hi guys, I just wanted to comment that I had fully working installation of Python until yesterday. I did brew upgrade and sqlite got upgraded. Now, I get the same error as peterlewis: ImportError: No module named _sqlite3. I have tried doing brew uninstall python and brew install python, but I haven't been able to get it to work again.

Contributor

Yep. Sorry it's broken. Fix soon. Meanwhile a manual brew link sqlite and the install python will work.

Cool, I just wanted to make sure it was known that this was breaking previously working installations (since the previous posts seem to all be talking about errors with fresh installs).

By the way, I implemented the tweak to python.rb suggested by staticfloat and samueljohn and then ran brew install python again, and sqlite seems to be working again.

I changed the lines staticfloat mentioned to:

  cflags = "CFLAGS=-I#{HOMEBREW_PREFIX}/include -I#{Formula.factory('sqlite').opt_prefix}/include"
  ldflags = "LDFLAGS=-L#{HOMEBREW_PREFIX}/lib -L#{Formula.factory('sqlite').opt_prefix}/lib"
@samueljohn samueljohn added a commit to samueljohn/homebrew that referenced this issue Jan 16, 2013
@samueljohn samueljohn python: Explicitly add sqlite to cflags/ldflags
Fixes #15300 because sqlite is keg_only since a few days.
660727b
Contributor

I opened a pull request to fix the "sqlite not found"-bug. Hope it gets pulled soon. Sorry for any inconvenience!

@adamv adamv closed this in a737465 Jan 16, 2013

You rock samueljohn! Thank you so much for your help towards this commit! :) Everything is running slick as a whistle again! :)

I hit the failure to build with sqlite problem with an install of homebrew from about 7 days ago specifically because it's needed for iPython history support. 18 Jan 2013 Ran brew update and brew doctor, brew uninstall python, followed by brew install python and both import sqlite3 is good and iPython runs without complaining that there's no sqlite so can confirm fixed for OSX 10.8.2 on a clean install. Many Thanks and keep up the good work.

I'm running into what I think is the same issue for python3 formula. Is there any plan to fix this as well?

H

On 21 Jan 2013, at 22:07, Pete Browne notifications@github.com wrote:

I'm running into what I think is the same issue for python3 formula. Is there any plan to fix this as well?


Reply to this email directly or view it on GitHub.

Hi,
apologies for the previous a finger slip on one of touch things. There's a little more wrong with the formula than just getting it to build with sqlite support for iPython as import foundation also fails. From memory, it's 4 years or so since I was regularly building Python from source, there's a choice on OSX between building with Foundation support Tkinter support or another widowing toolkit of choice. I'm going to refresh my memory on the options then pull the formula and have a go at it prolly over the weekend at some point.

On 22 Jan 2013, at 00:07, Edward Hartley ed_hartley@mac.com wrote:

H

On 21 Jan 2013, at 22:07, Pete Browne notifications@github.com wrote:

I'm running into what I think is the same issue for python3 formula. Is there any plan to fix this as well?


Reply to this email directly or view it on GitHub.

Contributor

@petebrowne, thanks, I am going to fix that soon.

@ReaddyEddy ImportError: No module named foundation ?
What do you mean by foundation? Does the system Python support that? I think Python only ships with Tk bindings by default. However, you can brew install wxwidgets or related to get additional.

@norioxkimura norioxkimura added a commit to norioxkimura/homebrew that referenced this issue Jan 26, 2013
@samueljohn @norioxkimura samueljohn + norioxkimura python: Explicitly add sqlite to cflags/ldflags
Fixes #15300 because sqlite is keg_only since a few days.

Closes #17120.

Signed-off-by: Adam Vandenberg <flangy@gmail.com>
89d3ee1

@ReaddyEddy @petebrowne Had the same issue with python3, resolved by doing a brew edit python3 and applying cflags and ldflags changes in this diff for the python 2.7 formula:

norioxkimura@89d3ee1

Try installing from this maybe: https://gist.github.com/4651708

Contributor
rburhum commented Feb 7, 2013

Sadly, I could not get uninstalling and reinstalling python did not work for me. I still get an error that sqlite is not found.

  (pycharm_repro)rburhum@peru.local ~/src/pycharm_repro/pycharm_repro $ python manage.py test
    Traceback (most recent call last):
      File "manage.py", line 10, in <module>
        execute_from_command_line(sys.argv)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
        utility.execute()
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
        klass = load_command_class(app_name, subcommand)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/management/__init__.py", line 69, in load_command_class
        module = import_module('%s.management.commands.%s' % (app_name, name))
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
        __import__(name)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django_nose/__init__.py", line 4, in <module>
        from django_nose.runner import *
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django_nose/runner.py", line 18, in <module>
        from django.core.management.commands.loaddata import Command
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 13, in <module>
        from django.core import serializers
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/serializers/__init__.py", line 21, in <module>
        from django.core.serializers.base import SerializerDoesNotExist
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/core/serializers/base.py", line 7, in <module>
        from django.db import models
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/db/__init__.py", line 40, in <module>
        backend = load_backend(connection.settings_dict['ENGINE'])
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/db/__init__.py", line 34, in __getattr__
        return getattr(connections[DEFAULT_DB_ALIAS], item)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/db/utils.py", line 92, in __getitem__
        backend = load_backend(db['ENGINE'])
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/db/utils.py", line 24, in load_backend
        return import_module('.base', backend_name)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
        __import__(name)
      File "/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 31, in <module>
        raise ImproperlyConfigured("Error loading either pysqlite2 or sqlite3 modules (tried in that order): %s" % exc)
    django.core.exceptions.ImproperlyConfigured: Error loading either pysqlite2 or sqlite3 modules (tried in that order): dlopen(/Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/lib-dynload/_sqlite3.so, 2): Library not loaded: /usr/local/lib/libsqlite3.0.8.6.dylib
      Referenced from: /Users/rburhum/.virtualenvs/pycharm_repro/lib/python2.7/lib-dynload/_sqlite3.so
      Reason: image not found

I also tried removing all homebrew packages and reinstalling them to be sure by installing sqlite first, then python, then the rest. Removing ".virtualenv" directory and recreating it. No luck.

The only thing that fixes it is to do brew link --force sqlite which I realize it may defeat the purpose of having it keg-only, but at least it makes my system usable again.

Contributor

@RBURHUM I assume you did a brew update in the last 22 days, right?

It should have been fixed by explicitly adding the opt_prefix of sqlite to the flags as you can see in this line: https://github.com/samueljohn/homebrew/blob/660727bd5d71efea260df266d2ca55238f739448/Library/Formula/python.rb#L183

You are running from inside a virtualenv, can you test without the virtualenv if import sqlite3 in the /usr/local/bin/python interpreter (or whereever you have installed homebrew to) succeeds?

Contributor
rburhum commented Feb 7, 2013

@samueljohn I did run brew update and brew upgrade yesterday :)

You are right though. It works outside of virtualenv, and inside of virtualenv it is broken.

Yesterday I removed ~/.virtualenvs yesterday and recreate it after logging in with a fresh upgraded terminal, the virtualenvs dir got recreated. Check out this output:

Works outside virtualenv

rburhum@peru.local ~ $ python -c "import sqlite3"

Create virtualenv - notice error with sitecustomize. Try to import sqlite3

rburhum@peru.local ~ $ mkvirtualenv test
'import sitecustomize' failed; use -v for traceback
New python executable in test/bin/python
Installing setuptools............done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test/bin/get_env_details
(test)rburhum@peru.local ~ $ python -c "import sqlite3"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: dlopen(/Users/rburhum/.virtualenvs/test/lib/python2.7/lib-dynload/_sqlite3.so, 2): Library not loaded: /usr/local/lib/libsqlite3.0.8.6.dylib
  Referenced from: /Users/rburhum/.virtualenvs/test/lib/python2.7/lib-dynload/_sqlite3.so
  Reason: image not found

Force link sqlite and run import again inside virtualenv

(test)rburhum@peru.local ~ $ brew link -f sqlite
Linking /usr/local/Cellar/sqlite/3.7.15.1... 9 symlinks created
(test)rburhum@peru.local ~ $ python -c "import sqlite3"
(test)rburhum@peru.local ~ $ 

Works fine that time.

I guess it is due to sitecustomize then?

This is the relevant portion of my bashrc file:

# Setting up the VirtualEnv
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python2.7
#export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages'
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true

if [[ -r /usr/local/share/python/virtualenvwrapper.sh ]]; then
    source /usr/local/share/python/virtualenvwrapper.sh
else
    echo "WARNING: Can't find virtualenvwrapper.sh"
fi
Contributor

Thanks for the report. So workaround for now: brew link --force sqlite.

I will put this on my todo list.

Contributor
rburhum commented Feb 7, 2013

Sure. Thanks to you @samueljohn

Contributor

I guess this is because during the creation of a virtualenv, the python stdlib (including the sqlite wrappers) are copied into the venv.

Can somebody test if creating a new virtualenv with the sqlite keg_only (and _un_linked), works and import sqlite3 works in that python within the venv? For this test to be meaningful, brew test python outside of the venv should succeed. If not, please brew rm python && brew install python.

Contributor
rburhum commented Feb 13, 2013
rburhum@peru.local ~ $ python
Python 2.7.3 (default, Feb  6 2013, 18:10:34) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.24)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> quit()
rburhum@peru.local ~ $ brew unlink sqlite
Unlinking /usr/local/Cellar/sqlite/3.7.15.2... 9 links removed
rburhum@peru.local ~ $ brew test python
Testing python
==> /usr/local/Cellar/python/2.7.3/bin/python -c import sqlite3
==> /usr/local/Cellar/python/2.7.3/bin/python -c import Tkinter; root = Tkinter.
rburhum@peru.local ~ $ mkvirtualenv test_sqlite
'import sitecustomize' failed; use -v for traceback
New python executable in test_sqlite/bin/python
Installing setuptools............done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test_sqlite/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test_sqlite/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test_sqlite/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test_sqlite/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/rburhum/.virtualenvs/test_sqlite/bin/get_env_details
(test_sqlite)rburhum@peru.local ~ $ brew test python
Testing python
==> /usr/local/Cellar/python/2.7.3/bin/python -c import sqlite3
==> /usr/local/Cellar/python/2.7.3/bin/python -c import Tkinter; root = Tkinter.
(test_sqlite)rburhum@peru.local ~ $ python -c "import sqlite3"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: dlopen(/Users/rburhum/.virtualenvs/test_sqlite/lib/python2.7/lib-dynload/_sqlite3.so, 2): Library not loaded: /usr/local/lib/libsqlite3.0.8.6.dylib
  Referenced from: /Users/rburhum/.virtualenvs/test_sqlite/lib/python2.7/lib-dynload/_sqlite3.so
  Reason: image not found
(test_sqlite)rburhum@peru.local ~ $ brew link -f sqlite
Linking /usr/local/Cellar/sqlite/3.7.15.2... 9 symlinks created
(test_sqlite)rburhum@peru.local ~ $ python -c "import sqlite3"
(test_sqlite)rburhum@peru.local ~ $ 
Contributor

Hmmmm ... I can't reproduce. Make sure you brew unlink sqlite, brew rm python, brew install python.
And after that do: mkvirtualenv test_sqlite, and in the venv start python and import sqlite3.

~ ❯ mkvirtualenv test_sqlite
New python executable in test_sqlite/bin/python
Installing distribute...........................................................................................................................................................................................................................done.
Installing pip................done.
virtualenvwrapper.user_scripts creating .virtualenvs/test_sqlite/bin/predeactivate
virtualenvwrapper.user_scripts creating .virtualenvs/test_sqlite/bin/postdeactivate
virtualenvwrapper.user_scripts creating .virtualenvs/test_sqlite/bin/preactivate
virtualenvwrapper.user_scripts creating .virtualenvs/test_sqlite/bin/postactivate
virtualenvwrapper.user_scripts creating .virtualenvs/test_sqlite/bin/get_env_details
(test_sqlite)~ ❯ which python
~/.virtualenvs/test_sqlite/bin/python
(test_sqlite)~ ❯ python
Python 2.7.3 (default, Feb 11 2013, 17:27:17) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.24)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Contributor
rburhum commented Feb 13, 2013

Still same behavior. Full log here:

http://dpaste.com/hold/924165/

Contributor

Okay does the virtualenv cache the python stdlib somehow?

I deleted (due to other reasons my virtualenvs dir).

Test wise you could set the ENV var where your virtualenvs are located
to another dir. Then mkvirtualenv will create the stuff anew.

Just please don't delete your venv :-)

Contributor
rburhum commented Feb 20, 2013

Did this also, no luck :(

Contributor

Any idea?
In the brewed python the import works but not in the virtualenv? Even if you set the dir where the virtualenvs are stored differently and start over a new (for me this worked)?

What are your virtualenv ENV vars (from your shell's configuration script) like? Can you post them?

dashesy commented Feb 26, 2013

Just wanted to report success after following clsung's comment, I first noticed sqlite problem when running ipython. Sorry if this is a noob question, is there any reason that sqlite shared library cannot be built/added after Python is already installed as part of "brew install sqlite3"? does Python link to static library if sqlite3 is not properly present?

Contributor

Not sure.
But for sure, python remembers the path to the sqlite dylib it had
during install. Therefore reinstalling python is needed here.

tloredo commented Mar 19, 2013

I'm not sure if this is any help, but I encountered this problem a few months ago. IPython (which uses sqlite3 for history tracking) worked fine until I did a "brew update" and updated sqlite from 3.7.14 to 3.7.15.1. I reverted to the previous formula, returning to 3.7.14, and all was fine again. I presumed this indicated a problem with 3.7.15 and not with homebrew.

Today I did a "brew update," and unlinked sqlite and then installed the current version. IPython gave a missing sqlite warning. I uninstalled python and re-installed it. IPython now works fine. So indeed the problem does appear to have been fixed. I'm running OS 10.7.5.

Contributor

Thanks for the round up. Absolutely correct. I made sqlite "keg_only"
due to other build issues. I oversaw that python needs an extra fix to
find the sqlite in the Cellar then (python is a bit non-conform). The
fix has been added to the python formula a bit later. But people need
to re-brew python, as you found out. We don't have a mechanism to tell
that a certain formulae need to be removed and brewed again (even if
no version bump has occurred).

Sorry for the trouble.

@xu-cheng xu-cheng locked and limited conversation to collaborators Feb 16, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.