Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keep old packages #56

Closed
MWigger opened this issue Jun 13, 2017 · 12 comments
Labels
bug

Comments

@MWigger
Copy link

@MWigger MWigger commented Jun 13, 2017

Hello,
I have a little struggle with conda-mirror:
We use environments where we sometimes define package versions.
Unfortunately conda-mirror deleted one of them, since it was "to old"
Is there a way to keep old packages, or even better: have a blacklist of packages not to remove?

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 13, 2017

Hi @MWigger. Thanks for the issue report. What version of conda-mirror are you running?

Unfortunately conda-mirror deleted one of them, since it was "to old"

I suspect what happened was that conda-mirror deleted a package that you had locally because it was removed from the upstream repository. There is currently no mechanism that checks the age of package. There are four conditions where a package will be removed.

  1. Package was deleted from the upstream repository
  2. Package is an invalid tarball
  3. Package fails the md5 checksum
  4. Package fails the sha256 checksum

If you were mirroring the defaults channel or the conda-forge channel, I can check my own logs at that time to see if I can help debug this issue.

Can you let me know which package was removed and the date that you noticed it? If you're not comfortable posting that information to github, please feel free to send me an email at eric.dill@maxpoint.com and I can help you debug this in a 1:1 conversation.

@MWigger

This comment has been minimized.

Copy link
Author

@MWigger MWigger commented Jun 13, 2017

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 13, 2017

Are you logging the output of conda-mirror? Whenever it removes packages it notes why. What is the upstream channel that you are mirroring?

I see these errors in my logs for the anaconda channel:

WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/osx-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...
WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/win-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...

FWIW conda-mirror will attempt to redownload packages every time you run it. You should be able to run conda-mirror again and it will attempt to download again any packages that failed validation the last time you ran it.

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 13, 2017

What I neglected to add to the last comment is that I currently have mkl-2017.0.1-0 locally. It did fail once, but has not failed in the last 3 or so weeks.

@MWigger

This comment has been minimized.

Copy link
Author

@MWigger MWigger commented Jun 13, 2017

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 13, 2017

Yes. Any package removal gets dumped as warning level which is -v

@MWigger

This comment has been minimized.

Copy link
Author

@MWigger MWigger commented Jun 13, 2017

@MWigger

This comment has been minimized.

Copy link
Author

@MWigger MWigger commented Jun 14, 2017

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 14, 2017

Thanks for the updated information @MWigger .

There are two distinct pieces happening there.

The first is the expected behavior of conda-mirror removing packages locally that have been removed upstream. The intent of conda-mirror is to be a complete mimic of what is available upstream. It is unfortunate that upstream removed a package that you were depending on locally. That being said, when packages are removed from upstream there is usually a good reason for this. I have zero control over or insight into the reasons for package removal from upstream.

The EOFError is something that I need to be catching. That's a new bug. Thanks for finding that 😀

@ericdill ericdill added the bug label Jun 14, 2017
@MWigger

This comment has been minimized.

Copy link
Author

@MWigger MWigger commented Jun 15, 2017

Well, I just saw thast for example for the cffi package there is a 1.7.0 version on the coda-forge channel: https://anaconda.org/conda-forge/cffi/files

So it might be a different issue

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Jun 16, 2017

I have seen similar package removals for 'failed size test'. I dug in to a couple of these occurances and what I found was that the package, let's take cffi-1.7.0-py27_0.tar.bz2 for example, had been deleted from the repo that I was mirroring and re-uploaded with one or more pieces of metadata changed. I think one time I looked, a license file had been added to the package. Since this does not affect the run time behavior of the package, bumping the build number (to cffi-1.7.0-py27_1.tar.bz2) was likely deemed unnecessary. Indeed only people that are maintaining their own local copy of the entire upstream repository would notice.

The reason why re-uploading a package with different metadata will cause conda-mirror to fail the size test is that the new uploaded package on upstream has a different size. The repodata.json file for that upstream repo has already been updated for the packages new size so when I compare the expected size in the upstream repodata.json with the size of the package that has been mirrored locally, they do not match. Because I treat the upstream repodata.json as the source of truth, that means the local package is now "incorrect" and it get's removed. Then the package that was just removed because of its "incorrect" size will be redownloaded from upstream.

@ericdill

This comment has been minimized.

Copy link
Contributor

@ericdill ericdill commented Aug 7, 2017

Closed by #58

@ericdill ericdill closed this Aug 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.