Import error when running dxr through mod_wsgi | 404 when trying to access files with built-in web server

[averi]

I did a fresh install of dxr today on a Fedora 20 machine, it worked smoothly but I encountered a few issues:

Trying to run dxr through mod_wsgi results in a "ImportError: No module named site" on error_log, the vhost:

WSGIPythonHome /home/gnomeweb/dxr_venv

<VirtualHost *:80>
    ServerName dxr-test.gnome.org

    CustomLog /var/log/httpd/dxr-test.gnome.org/access_log combined
    ErrorLog /var/log/httpd/dxr-test.gnome.org/error_log

    # Serve static resources, like CSS and images, with plain Apache:
    Alias /static/ /home/gnomeweb/dxr/dxr/static/

    # We used to make special efforts to also serve the static pages of
    # HTML-formatted source code from the tree via plain Apache, but that
    # tangle of RewriteRules saved us only about 20ms per request. You can do
    # it if you're on a woefully underpowered machine, but I'm not maintaining
    # it.

    # Tell this instance of DXR where its target folder is:
    SetEnv DXR_FOLDER /home/gnomeweb/dxr_output/

    SetEnv PYTHONHOME /home/gnomeweb/dxr_venv

    WSGIScriptAlias / /home/gnomeweb/dxr/dxr/wsgi.py
</VirtualHost>

<Directory /home/gnomeweb/dxr/dxr>
   Require all granted
</Directory>

Running dxr from dxr-serve.py works just fine but another issue arises.

All the files are indexed properly and searching for code through the search bar works as expected and every single file content is properly showed when a string matches the search but I can seem to access the files content when clicking on them, for example:

http://dxr-test.gnome.org/git-test/source/gtk+ works OK
http://dxr-test.gnome.org/git-test/source/gtk+/AUTHORS returns a 404

The directories are accessible by the web server and I can't seem to find any permission error on the setup, my dxr_output folder only shows directories but no files at all, an excerpt of a tree run:

|-- build
|   `-- win32
|       |-- crypt
|       |-- vs10
|       |-- vs11
|       |-- vs12
|       `-- vs9
|-- demos
|   |-- gtk-demo
|   |   `-- data
|   |       |-- 16x16
|   |       |-- 22x22
|   |       |-- 24x24
...

Thanks for all the help in advance!

[abbeyj]

On Sun, Aug 10, 2014 at 5:55 PM, Andrea Veri notifications@github.com
wrote:

WSGIPythonHome /home/gnomeweb/dxr_venv

Do you have this line in the vhost section? According to
http://dxr.readthedocs.org/en/latest/deployment.html#apache-and-mod-wsgi it
won't work there.

Running dxr from dxr-serve.py works just fine but another issue arises.

1. All the files are indexed properly and searching for code through
   the search bar works as expected and every single file content is properly
   showed when a string matches the search but I can seem to access the files
   content when clicking on them, for example:

http://dxr-test.gnome.org/git-test/source/gtk+ works OK
http://dxr-test.gnome.org/git-test/source/gtk+/AUTHORS returns a 404

The directories are accessible by the web server and I can't seem to find
any permission error on the setup, my dxr_output folder only shows
directories but no files at all, an excerpt of a tree run:

You should have one file in the output directory for every input file. If
you have a file called foo.cpp in your source tree there should be a
corresponding foo.cpp.html in your output directory. If that's not there
then it looks like something went wrong during indexing. Try to run the
indexer again or check the log files from the last run for any problems.

[averi]

WSGIPythonHome /home/gnomeweb/dxr_venv is outside from the vhost itself as it should be.

About the files indexing seems the problem is really there and no html file is created at all for each of them. Directories do have a .dxr-directory-index.html file but no other files are showing up there. Additionally /tmp/dxr-temp/logs/git-test/build.log shows pretty much nothing and the documentation does not seem to provide log levels to increase debugging messages. I'm also wondering if this is really a permissions problem as directories are indexed correctly (and they have 0755 permissions) while files aren't. (with 0644 permissions, and surely the files don't need to be executable themselves /me thinks)

I'm clueless : /

[abbeyj]

Can you copy-and-paste the output from running the indexer as well as the
contents of build.log?

[averi]

Indeed!

Indexer:

Generating target folder
Creating tables
Indexing files from the 'git-test' tree
(finished in 0:01:05.200946)
Building the 'git-test' tree
Build command for 'git-test' failed, exited non-zero.
Log follows:
    | make: *** No targets specified and no makefile found.  Stop.

build.log

make: *** No targets specified and no makefile found.  Stop.

With default build_command:

build_command = make clean; make -j $jobs

Is this probably the cause of the problem? i.e not building the sources properly?

[abbeyj]

Yes, that looks like the problem. Note that the build_command gets
executed from the object_folder not the source_folder. If your source code
is designed to be built in-tree then you can add "cd $source_folder && " to
the beginning of your build_command. Otherwise you have to do whatever is
needed to set up the object_folder with a makefile. For autoconf-based
setups you'd want something like "$source_folder/configure && ".

[averi]

Thanks James! I have two more questions though:

1. does this mean we would have to specify the build commands for each of the repositories we are going to serve through dxr? (in the case the build system differs for example)
2. does this mean all the build dependencies for each of the hosted repositories has to be installed on the machine? (answer to 2. should be a strict consequence of 1.'s answer as otherwise all the builds will just fail)

[averi]

About the missing HTML files creation: dxr-build.py was failing cause of a missing plugin configuration key. Adding the missing key wasn't enough as the build kept feeling, so disabled_plugins = buglink under the DXR configuration section was enough.

Additionally seems there is no current way to index repositories available outside GitHub, the current workaround would be: disabled_plugins omniglot.

That fixed the build for me.

[erikrose]

It's git repos outside GitHub that are the problem, specifically. I've opened a ticket to fix that: https://bugzilla.mozilla.org/show_bug.cgi?id=1052572. Thanks for being persistent!