Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--skip-existing seems not to work with some Nexus version #693

Closed
cyrilou242 opened this issue Sep 15, 2020 · 4 comments · Fixed by #694
Closed

--skip-existing seems not to work with some Nexus version #693

cyrilou242 opened this issue Sep 15, 2020 · 4 comments · Fixed by #694
Milestone

Comments

@cyrilou242
Copy link
Contributor

Environment

  1. Your operating system:
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G13035
  1. Version of python you are running:
Python 3.6.6
  1. How did you install twine? Did you use your operating system's package manager or pip or something else?
pip
  1. Version of twine:
twine version 3.2.0 (pkginfo: 1.5.0.1, requests: 2.24.0, setuptools: 49.6.0,
requests-toolbelt: 0.9.1, tqdm: 4.31.1)
  1. Which package repository are you targeting?
Nexus Repository OSS (https://www.sonatype.com/nexus-repository-oss)
Version | 3.25.1-04
Edition | OSS
Build Revision | d5c8e51f950fa30ce691e52998479b09ec888a2e
Build Timestamp | 2020-07-30-1345-34270

Issue

The --skip-existing flag that is supposed to ignore upload errors when a package is already existing in the repository is not working with my configuration: Nexus Repository OSS: 3.25.1-04.

When trying to upload a package that is already existing, I had a look at what's happening in the skip_upload function:

def skip_upload(

status, reason and text values are obtained from the response

In my case, I have:
status: 400
reason: bad request
text:

<!doctype html>
<html lang="en">
<head>
  <title>400 - nexus repository manager</title>
  <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
  <!--[if lt ie 9]>
  <script>(new image).src="https://[private.repo.io]/favicon.ico?3.25.1-04"</script>
  <![endif]-->
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-32x32.png?3.25.1-04" sizes="32x32">
  <link rel="mask-icon" href="https://[private.repo.io]/safari-pinned-tab.svg?3.25.1-04" color="#5bbad5">
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-16x16.png?3.25.1-04" sizes="16x16">
  <link rel="shortcut icon" href="https://[private.repo.io]/favicon.ico?3.25.1-04">
  <meta name="msapplication-tileimage" content="https://[private.repo.io]/mstile-144x144.png?3.25.1-04">
  <meta name="msapplication-tilecolor" content="#00a300">
  <link rel="stylesheet" type="text/css" href="https://[private.repo.io]/static/css/nexus-content.css?3.25.1-04"/>
</head>
<body>
<div class="nexus-header">
  <a href="https://[private.repo.io]">
    <div class="product-logo">
      <img src="https://[private.repo.io]/static/images/nexus.png?3.25.1-04" alt="product logo"/>
    </div>
    <div class="product-id">
      <div class="product-id__line-1">
        <span class="product-name">nexus repository manager</span>
      </div>
      <div class="product-id__line-2">
        <span class="product-spec">oss 3.25.1-04</span>
      </div>
    </div>
  </a>
</div>
<div class="nexus-body">
  <div class="content-header">
    <img src="https://[private.repo.io]/static/rapture/resources/icons/x32/exclamation.png?3.25.1-04" alt="exclamation point" aria-role="presentation"/>
    <span class="title">error 400</span>
    <span class="description">bad request</span>
  </div>
  <div class="content-body">
    <div class="content-section">
      repository does not allow updating assets: pypi-hosted
    </div>
      </div>
</div>
</body>
</html>

The skip_upload function is looking for a 400 and "updating assets" in the reason string, but in my case "updating assets" is not in reason but in text.

I would suggest to change

or (status == 400 and "updating asset" in reason)

with something like

        # Nexus Repository OSS (https://www.sonatype.com/nexus-repository-oss)
        or (status == 400 and ("updating asset" in reason  or "updating asset" in text))

Steps to Reproduce

I have not tried to replicate with a local nexus repo.

@sigmavirus24
Copy link
Member

in text would be ridiculously slow. Would be so nice for these repositories to not just break things arbitrarily.

@cyrilou242
Copy link
Contributor Author

cyrilou242 commented Sep 15, 2020

Thanks for the quick answer and I definitely agree with the point on repositories not breaking things.

Given that the boolean evaluation is lazy the above code would only impact uploads returning a 400 and not having "updating asset" in reason.
Do you think it's still too impactful on the run time ?

For my example, time seems to be ok:

import  timeit

text = """
<!doctype html>
<html lang="en">
<head>
  <title>400 - nexus repository manager</title>
  <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
  <!--[if lt ie 9]>
  <script>(new image).src="https://[private.repo.io]/favicon.ico?3.25.1-04"</script>
  <![endif]-->
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-32x32.png?3.25.1-04" sizes="32x32">
  <link rel="mask-icon" href="https://[private.repo.io]/safari-pinned-tab.svg?3.25.1-04" color="#5bbad5">
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-16x16.png?3.25.1-04" sizes="16x16">
  <link rel="shortcut icon" href="https://[private.repo.io]/favicon.ico?3.25.1-04">
  <meta name="msapplication-tileimage" content="https://[private.repo.io]/mstile-144x144.png?3.25.1-04">
  <meta name="msapplication-tilecolor" content="#00a300">
  <link rel="stylesheet" type="text/css" href="https://[private.repo.io]/static/css/nexus-content.css?3.25.1-04"/>
</head>
<body>
<div class="nexus-header">
  <a href="https://[private.repo.io]">
    <div class="product-logo">
      <img src="https://[private.repo.io]/static/images/nexus.png?3.25.1-04" alt="product logo"/>
    </div>
    <div class="product-id">
      <div class="product-id__line-1">
        <span class="product-name">nexus repository manager</span>
      </div>
      <div class="product-id__line-2">
        <span class="product-spec">oss 3.25.1-04</span>
      </div>
    </div>
  </a>
</div>
<div class="nexus-body">
  <div class="content-header">
    <img src="https://[private.repo.io]/static/rapture/resources/icons/x32/exclamation.png?3.25.1-04" alt="exclamation point" aria-role="presentation"/>
    <span class="title">error 400</span>
    <span class="description">bad request</span>
  </div>
  <div class="content-body">
    <div class="content-section">
      repository does not allow updating assets: pypi-hosted
    </div>
      </div>
</div>
</body>
</html>
"""

setup = 'text = """{}""";'.format(text)
number=100000
r = timeit.timeit('"updating asset" in text', setup=setup, number=number)
print(r/number)

--> around 1e-06 seconds on a correct macbook (not very precise and more powerful that a ci machine indeed)

400 + "bad request" is obviously not enough.
Also in text is already in use for Artifactory:

or (status == 403 and "overwrite artifact" in text)

@sigmavirus24
Copy link
Member

I mean we might as ewll

@cyrilou242
Copy link
Contributor Author

cyrilou242 commented Sep 17, 2020

This is fixed by #694.
I let a member close (depends if you want to close now or at next release)

@bhrutledge bhrutledge added this to the 3.3.0 milestone Oct 4, 2020
@bhrutledge bhrutledge linked a pull request Oct 4, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants