Testing catalog build locally doesn't mistakenly retain old games

[edwardchalstrey1]

Issues closed by this PR

• Closes [Bug]: old catalog_data not cleared out on a pip install #920

Description of the changes in this PR

This PR fixes stale catalog_data/ files persisting after pip install -e . by swapping the lookup priority in src/pygambit/catalog.py: the repo's live (and crucially differently named) catalog/ directory is now preferred over the installed catalog_data/, with catalog_data/ as the fallback for deployed (non editable install) packages, which will lack the catalog/ dir.

This ensures catalog changes are always picked up immediately in a editable install development checkout without reinstalling, and eliminates divergence between local and CI environments caused by leftover files from previous installs.

Note: non-editable installs with pip install . should already replace the catalog data correctly due to this logic in setup.py:

class GambitBuildPy(setuptools.command.build_py.build_py):
    """Extend `build_py` to copy the catalog games data into the build library."""

    def run(self) -> None:
        super().run()

        catalog_source = pathlib.Path("catalog")
        catalog_target = pathlib.Path(self.build_lib) / "pygambit/catalog_data"
        if catalog_target.exists():
            # Wipe the old catalog_data/ entirely
            shutil.rmtree(catalog_target)
        catalog_target.mkdir(exist_ok=True, parents=True)
        # Fresh copy of catalog_data/ from catalog/
        shutil.copytree(catalog_source, catalog_target, dirs_exist_ok=True)

There is also a new clean_stale_img_files function added which gets rid of anything in catalog/img which doesn't match the structure of catalog

How to review this PR

• Test that when making changes to the catalog in an editable install and then doing pygambit.catalog.games() the changes are picked up
• When running the update script, any files and folders in catalog/img from old games that were removed are removed

[rahulsavani]

Unfortunately, this does not work yet.

My best guess is that the repo's live catalog/img directory contains everything from previous versions and is not getting cleared out. If that is indeed the problem, the fix might be as simple as a directory clear when running

python build_support/catalog/update.py --build --regenerate-images

?

[rahulsavani]

Not working yet -- see my comment above.

[edwardchalstrey1]

Unfortunately, this does not work yet.

My best guess is that the repo's live catalog/img directory contains everything from previous versions and is not getting cleared out. If that is indeed the problem, the fix might be as simple as a directory clear when running

python build_support/catalog/update.py --build --regenerate-images

?

I think ensuring when running the update script that the structure of catalog/img matches catalog is a check worth adding anyway, and actually should be done regardless of whether --build (for updating catalog.am) or --regenerate-images (for forcing the images to be remade for games which already have them as well as new ones) is present.

[edwardchalstrey1]

@rahulsavani See the top comment - on further inspection, the 2 things that were broken and fixed by this PR are editable installs not picking up live changes properly and old images not being deleted. However, if you are doing a non editable pip install . each time before running the update script, this should already picking up changes correctly before this PR (see explanation above). So either way you shouldn't experience any more issues - let me know if you do!

[rahulsavani]

Unfortunately, Ed, this still doesn't seem to work for me. At this stage could you perhaps create a MWE of the problem locally so you can observe it for yourself and see if this fixes it for you?

[edwardchalstrey1]

Unfortunately, Ed, this still doesn't seem to work for me. At this stage could you perhaps create a MWE of the problem locally so you can observe it for yourself and see if this fixes it for you?

What's the error you're getting? I've tested and not seeing any problems at this point

[rahulsavani]

STARTING STATE OF catalog/img:

$ ls catalog/img
bagwell1995.ef bagwell1995.svg conf journals reiley2008 vonstengel2022
bagwell1995.pdf bagwell1995.tex gilboa1997 myerson1991 selten1975 vonstengelforges2008
bagwell1995.png books jakobsen2016 nau2004 shapley1974 watson2013

ACTUAL STATE OF catalog/:

$ls catalog
books conf img journals misc

PROCESS I RUN

$ pip install .
$ python build_support/catalog/update.py --build --regenerate-images

Removed 144 stale item(s) from ???/gambit/catalog/img

BUT

Things seem to get removed from /catalog/img but then put back. After the above completes:

$ ls catalog/img
bagwell1995.ef bagwell1995.svg conf journals reiley2008 vonstengel2022
bagwell1995.pdf bagwell1995.tex gilboa1997 myerson1991 selten1975 vonstengelforges2008
bagwell1995.png books

I.e. the directory seems to get fully emptied and then filled up incrementally with all the original stuff, even though half of it is not in the directory catalog itself!

And the final catalog table has everything just like at the start of the process.

[edwardchalstrey1]

Ok thanks, it might be that your pygambit install in your python environment is still retaining old games in the package data that should be removed when you redo the pip install, (like bagwell1995.efg which is replaced by journals/geb/bagwell.efg), or it might be a bug with the images although I can't replicate either - can you check if you do in Python:

import pygambit
pygambit.catalog.games()

after having done your recent pip install, does it have the expected games or still retain the old ones e.g. bagwell1995