neither package_data or data_files get installed

[ericfrederich]

I have tried this on Python 2.7.5 and Python 3.4.3 and am unable to see package_data.dat or data_file installed anywhere when I do "python setup.py install"
If I change to distutils it works fine.

[qwcode]

can you provide details with a paste of your output

[qwcode]

ok, I see. it works when installing (using pip or easy_install) from a built sdist, but not using "python setup.py install" from a checkout. I wasn't aware. will look into it.

[andyneff]

I think this is related to @ericfrederich issue?

Test 1) If I do python setup.py install I get an egg containing both package_data.dat and data_file, similarly if I set zip_safe to False, I have those files both in the sample directory.

Test 2) I create a wheel using pip wheel --wheel-dir=. sampleproject and when I do pip install sample-1.2.0-py2.py3-none-any.whlthe my_data\data_file works, and sample_data.dat

Test 3) But when I create a sdist (python setup.py sdist), the zip always contains package_data.dat (even if I comment out the package_data section), and never contains data_file.

error: can't copy 'data\data_file': doesn't exist or not a regular file

Test 4) Now if I remove package_data.dat from MANIFEST.in, it only includes the package_data as I would have expected...I guess when creating an sdist, these files are ALWAYS added even after python 2.6 due to MANIFEST.in

I guess this is all related to the same problem that's been plaguing sdist vs bdist forever http://blog.codekills.net/2011/07/15/lies,-more-lies-and-python-packaging-documentation-on--package_data-/ Although I don't know how much of that is still accurate or translates to setuptools, but I DO know the same problems appear to be happening...It's just this sample doesn't appear to work for both... YET?

Python 2.7.9 Windows 64-bit and Linux 64-Bit
Setuptools 15.2 and 16.0 (And I even tried 1.0 for a few tests, nothing really changed)

[ionelmc]

Neither package_data or data_files should be part of any "regular" project. They are ridiculously hard to get right. data_files behaves inconsistently across tools. I consider those anti-patterns in packaging [1]. Having a correct MANIFEST.in and include_package_data=True is most always the right choice.

[1] If you have the patience watch this

[qwcode]

yea @ionelmc , I think the sample project and the distribution tutorial needs some rehaul in this department. just recommend the easiest thing (like in your video), and maybe have an advanced section to cover all the caveats etc... that are so hard to remember.

[lrq3000]

Using include_package_data=True fixed the issue for me. Thank's @ionelmc !

[stuaxo]

@ionelmc what is the way to include some data with a project, that will get installed into the virtualenv (and allow installation using pip) ?

[ionelmc]

@stuaxo MANIFEST.in + include_package_data=True is my default approach. I'm not really sure what you're asking me, describe your situation more.

[stuaxo]

Ah, was messing with this too late and had broken my MANIFEST.in. Cheers for the pointer.

[laike9m]

@ionelmc MANIFEST.in + include_package_data=True as best practice, So true! I spent hours debugging what's wrong with package_data and finally solved it by adding a MANIFEST.

[stuaxo]

OT: Setuptools is such a mess, and the defaults mean I have to look up how to do this every year or so.

I'm glad work is being done on it, but having delved inside, I think we'd be better off chucking it out and using something else at this point.

[fduxiao]

When will this be fixed. You are justing wasting others' time, which is a kind of murder!

[stuaxo]

@fduxiao it's funny, as I only package something about once a year, I always, without fail - forget how to add non .py files to my python package, python packaging is almost always painful in some way.

[fduxiao]

@stuaxo In my situation, I use cython to accelerate my web application, which requires non python templates and will be used further by other projects, and I found the only way that works is putting an empty init.py into the templates dir and tell cython not to handle it (they are found automatically). It cost me more then 3 hours to find it.
I think it will be great if there's a simple way to add non python files.

[nicktimko]

@fduxiao is the workaround of MANIFEST.in (docs here) and setup(..., include_package_data=True) not working for you?

[fduxiao]

@nicktimko See my fork here. I'm not sure whether I write the MANIFEST.in correctly or not.

[nicktimko]

Saying graft data would be slightly simpler than recursive-include data *, but does it not work now?

StackOverflow might be a better forum for such a question than here.

[fduxiao]

@nicktimko It never works if my package has only compile modules (You see I just cythonized the modules and the data just fail to be installed). The conclusion I can get from stackoverflow is that it's only a dirty lie. I'm not asking a question. I'm saying one could not put package data for a package of compile modules. It may be caused by the fact that setup never thinks compiled modules as packages (they are under some folder) and I cannot specify the folder is actually a package, so can anyone find the reason and fix it?

[stuaxo]

@fduxiao I had to do the thing with a fake __init__.py but it feels like a hack, the kind that might just stop working some day.

[andreypanin]

Hello all,

Yesterday for the first time I faced a need to create a Python package. I didn't think it would be such a pain if it's supposed to have data files! I've skimmed through the documentation, but it's so diverse and inaccurate that I can't wrap my head around it.

I set up a simple directory structure:

.
├── config.yml
├── MANIFEST.in
├── sample
│   ├── hello.py
│   └── __init__.py
└── setup.py

The contents of MANIFEST.in is:

include config.yml

The setup.py is:

from setuptools import setup

setup(
    name='sample',
    version='1.0',
    packages=['sample'],
    include_package_data=True,
)

When I do pip install . or python setup.py install, a directory is created under .../lib/python3.6/site-packages, but there's no config.yml that I hoped would be included there, hence I cannot access it from code.

What am I doing wrong? Any help is appreciated. Thank you.

PS: Have watched @ionelmc 's presentation and tried the following MANIFEST.in: graft <package_name> <data_dir> but to no avail.

[ionelmc]

@andreypanin config.yml is not inside the package named sample thus it doesn't qualify as package data, thus include_package_data=True won't have any effect. Move it inside the sample directory and use graft sample in the manifest (just that).

[KenTsui]

Is this problem solved? I can use package_data to include my image files successfully into my package.

Python 3.7.3 64bit
setuptools - 40.8.0

[pradyunsg]

I'm gonna go ahead and close this -- packagers should use MANIFEST.in and include_package_data=True to correctly choose which files get included.

You can look at the pip's setup.py and MANIFEST.in for a more concrete example.

If anyone has similar/other issues related to which files get included, please check if there's a related issue filed on setuptools' issue tracker, otherwise please file a new issue there. :)

Thanks everyone for chiming in here!

[pradyunsg]

To be clear, using a different build tool like flit can help avoid these setuptools specific issues.

[lrq3000]

With Python 3 I've found out that the MANIFEST.in is not honored anymore in the wheel, but it is in the tar.gz! Maybe this was always the case but I just noticed now.

Anyway, here is the fix: to force the wheel to honor the MANIFEST.in, you need to specify both include_package_data and packages.

If your Python module is inside a folder "mymodule", here is how the setup looks like:

setup( ...
    include_package_data=True,  # use MANIFEST.in to include config file
    packages=['mymodule'],  # to force the wheel to use the MANIFEST.in
)

If all your python scripts are at the root (and not inside a subfolder), use:

setup( ...
    include_package_data=True,  # use MANIFEST.in to include config file
    packages=['.'],  # to force the wheel to use the MANIFEST.in
)