Updating MANIFEST.in does not correctly update the package sdist creates

Originally reported by: spookylukey (Bitbucket: spookylukey, GitHub: spookylukey)

---

The behaviour of sdist depends on previous contents of MANIFEST.in, not just the current. This is not fixed even by running setup.py clean or setup.py clean --all (although this should not be necessary).

This is very surprising behaviour, and potentially dangerous too - if someone accidentally adds a MANIFEST.in rule that includes a file that must not be distributed and notice the problem, they would expect that removing the rule will remove the file, but it does not.

I've attached a bash script that demonstrates the problem.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/436

[miccoli]

I have been hit by this same bug. My current remedy is to delete the *.egg-info directory before running setup.py sdist.

[miccoli]

The problema appears to be in the logic with which the SOURCES.txt file is updated by python setup.py egg_info:

$ python setup.py egg_info
running egg_info
writing sample.egg-info/PKG-INFO
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
writing requirements to sample.egg-info/requires.txt
writing top-level names to sample.egg-info/top_level.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'

Files already in SOURCES.txt are preserved across subsequent runs of python setup.py egg_info. Could please one of the developers clarify if this is intended behaviour and why?

[miccoli]

See also

setuptools/setuptools/command/egg_info.py

Lines 560 to 570 in d4c215a

	def add_defaults(self):
	sdist.add_defaults(self)
	self.filelist.append(self.template)
	self.filelist.append(self.manifest)
	rcfiles = list(walk_revctrl())
	if rcfiles:
	self.filelist.extend(rcfiles)
	elif os.path.exists(self.manifest):
	self.read_manifest()
	ei_cmd = self.get_finalized_command('egg_info')
	self.filelist.graft(ei_cmd.egg_info)

where the actual reading of SOURCES.txt takes place:

        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

I cannot understand the reason for this logic.

[miccoli]

@jaraco I think I nailed down this old (annoying bug), but nobody cares of these old bugs. Should I reopen or provide a pull request?

[jaraco]

@miccoli: If this ticket describes the issue, it's still open. A PR would be most appreciated.

[miccoli]

@jaraco yes: this ticket is accurate. Since there is no discussion active, I assume that I should provide my own PR.

[PiDelport]

I think this change may have broken sdist installs with package data: #1016

[miccoli]

The reason for

        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

is now clear: in the develop tree (under SCM control) the files to be installed are determined from the SCM system (walk_revctrl()); in an sdist install the files are to be determined from the existing manifest file, which was previously generated in the develop tree + setuptools_scm

This logic is broken for packages that do not use setuptools_scm, for which there is no guarantee that the current manifest file (in the development tree) is correct.

Unfortunately my PR #1014 badly brokes this intended behaviour, so that pip install of sdist packages under SCM control and include_package_data=True is not more possible, see #1016 .