File size check when detecting changed inputs (Provide a mechanism to force cache rebuild)

[scdub]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

If builds are not deduplicated with the hash suffix, replacing package archives will not be detected by conda_index and it will provide out of date metadata. Checking on file size would be a relatively inexpensive way to detect this case, but an option to force cache recreation may be simpler to implement and not impose runtime costs for environments where hash suffixes are exclusively used.

Conda Info

active environment : base
    active env location : /home/user/miniconda
            shell level : 1
       user config file : /home/user/.condarc
 populated config files :
          conda version : 23.7.2
    conda-build version : 3.26.0
         python version : 3.9.16.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.27=0
                          __linux=6.2.9=0
                          __unix=0=0
       base environment : /home/user/miniconda  (writable)
      conda av data dir : /home/user/miniconda/etc/conda
  conda av metadata url : None
          package cache : /home/user/miniconda/pkgs
                          /home/user/.conda/pkgs
       envs directories : /home/user/miniconda/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/23.7.2 requests/2.31.0 CPython/3.9.16 Linux/6.2.9-x86_64-linode160 ubuntu/18.04.6 glibc/2.27
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Conda Config

==> envvars <==
allow_softlinks: False

==> cmd_line <==
debug: False
json: False

Conda list

No response

Additional Context

Removing the .cache directories forces a refresh, this wasn't immediately obvious to me migrating from the historical conda index

[dholth]

I'm surprised we don't already detect that kind of change. In these lines we store the path, mtime, and size. When we retrieve it we only check mtime though.

https://github.com/conda/conda-index/blob/main/conda_index/index/sqlitecache.py#L420-L489

We also try to copy those values over from the older many-small-json cache. https://github.com/conda/conda-index/blob/main/conda_index/index/convert_cache.py#L246-L251

[scdub]

Thanks for the details. Do you think we could safely add a check on the size during the comparison with fs.mtime != cached.mtime OR fs.size != cached.size OR cached.path IS NULL)? I rebuilt the index manually after I encountered this so I can't directly reproduce it currently, but I can try to do so and breakpoint into the code if that'd be helpful.

[dholth]

Yes that sounds like a good addition. We can easily test by updating the DB with bad mtimes or sizes to see what happens.

…

On Thu, Aug 24, 2023, 9:27 PM Shaun Walbridge ***@***.***> wrote: Thanks for the details. Do you think we could safely add a check on the size during the comparison <https://github.com/conda/conda-index/blob/ceb9cabd1c24beff750702093322fe480e0ecd90/conda_index/index/sqlitecache.py#L481> with (fs.mtime != cached.mtime OR fs.size != cached.size OR cached.path IS NULL)? I rebuilt the index manually after I encountered this so I can't directly reproduce it currently, but I can try to do so and breakpoint into the code if that'd be helpful. — Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABSZETWYSOS3UVUW5QI3BTXW75QDANCNFSM6AAAAAA3B2GYMA> . You are receiving this because you commented.Message ID: ***@***.***>

[scdub]

OK, I was able to observe this again: if you replace a package with an exact matching name to one that is already cached, e.g. graphviz-8.1.0-0.conda, the metadata will not be rewritten. I believe the suggested change will address that for this particular case of not using hash strings.

[dholth]

This excerpt from a changed-packages query checks md5 or sha256 sums, only when the upstream or 'fs' state has a non-empty checksum.

            WHERE fs.path LIKE :path_like AND
                (cached.path IS NULL OR
                    (LENGTH(fs.sha256) > 0 AND fs.sha256 != cached.sha256)
                    OR
                    (LENGTH(fs.md5) > 0 AND fs.md5 != cached.md5)
                )