Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traceback when publishing amazon.aws #52

Closed
mamercad opened this issue Jan 30, 2023 · 17 comments · Fixed by #72
Closed

Traceback when publishing amazon.aws #52

mamercad opened this issue Jan 30, 2023 · 17 comments · Fixed by #72
Assignees
Labels
bug Something isn't working

Comments

@mamercad
Copy link
Contributor

When trying to publish amazon.aws:==5.2.0, I'm seeing:

ERROR:galactory:Exception on /api/v2/collections/amazon/aws/ [GET]
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/api/v2/collections.py", line 47, in collection
    results = _collection_listing(repository, namespace, collection, scheme=scheme)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 162, in _collection_listing
    collections = collected_collections(repo, namespace, collection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 143, in collected_collections
    for c in discover_collections(repo, namespace=namespace, name=name, scheme=scheme):
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 107, in discover_collections
    collection_info = json.loads(props['collection_info'][0])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 482 (char 481)
@mamercad
Copy link
Contributor Author

I have a hunch ... looks like there's not enough metadata ... I don't see fqcn, namespace, name and more.

image

@mamercad
Copy link
Contributor Author

I'm hacking on this a bit locally, manually raised a RuntimeError to dump out the properties and they look ok:

RuntimeError: {'collection_info': '{"namespace": "amazon", "name": "aws", "version": "5.2.0", "authors": ["Ansible (https://github.com/ansible)"], "readme": "README.md", "tags": ["amazon", "aws", "cloud"], "description": null, "license": [], "license_file": "COPYING", "dependencies": {}, "repository": "https://github.com/ansible-collections/amazon.aws", "documentation": "https://ansible-collections.github.io/amazon.aws/branch/stable-5/collections/amazon/aws/index.html", "homepage": "https://github.com/ansible-collections/amazon.aws", "issues": "https://github.com/ansible-collections/amazon.aws/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc"}', 'namespace': 'amazon', 'name': 'aws', 'version': '5.2.0', 'fqcn': 'amazon.aws'}

@briantist briantist self-assigned this Jan 30, 2023
@briantist
Copy link
Owner

Hi @mamercad !

This happens during publish? The top of the traceback shows a GET request:

ERROR:galactory:Exception on /api/v2/collections/amazon/aws/ [GET]

And the rest of it shows reads (it's a JSON decoding issue that seems to have raised the exception in the first place.

So I'm wondering if there's a longer trace that might show more information on the sequence of events.

Also could you show me the snippet you added that shows the RuntimeError? Maybe do a git diff and post that so I can see it in context.

From this comment, are you seeing that only collection_info is in Artifactory as a property?

When looking at it in Artifactory, is it complete/valid JSON?

Also wondering if you still have that property fallback option enabled, or set it back to doing uploads in one shot.

@mamercad
Copy link
Contributor Author

...the "missing properties" might be a red herring; I've been digging all day and what I'm seeing are sporadic Artifactory timeouts.

@mamercad
Copy link
Contributor Author

When looking at it in Artifactory, is it complete/valid JSON?

Ooh, at least from the UI, it does not seem valid, it appears that it's truncated!

image

...maybe we're hitting a limit.

@mamercad
Copy link
Contributor Author

Still researching, here's a bug I found upstream which may be related.

@mamercad
Copy link
Contributor Author

No mention of property constraints here.

@mamercad
Copy link
Contributor Author

Yup, it's truncated:

❯ curl -s -X GET -H "X-JFrog-Art-Api: ${ARTIFACTORY_API_KEY}" "https://REDACTED/artifactory/api/storage/ansible-dev-local/amazon-aws-5.2.0.tar.gz?properties" | jq -r '.properties.collection_info[0]'    
{"namespace": "amazon", "name": "aws", "version": "5.2.0", "authors": ["Ansible (https://github.com/ansible)"], "readme": "README.md", "tags": ["amazon", "aws", "cloud"], "description": null, "license": [], "license_file": "COPYING", "dependencies": {}, "repository": "https://github.com/ansible-collections/amazon.aws", "documentation": "https://ansible-collections.github.io/amazon.aws/branch/stable-5/collections/amazon/aws/index.html", "homepage": "https://github.com/ansible-collections/amazon.aws", "issues": "https://github.com/ansible-collections/amazon.aws/issues

Versus:

❯ curl -s -X GET -H "X-JFrog-Art-Api: ${ARTIFACTORY_API_KEY}" "https://REDACTED/artifactory/api/storage/ansible-dev-local/cloud-common-2.1.1.tar.gz?properties" | jq -r '.properties.collection_info[0]' | jq
{
  "namespace": "cloud",
  "name": "common",
  "version": "2.1.1",
  "authors": [
    "Ansible (https://github.com/ansible)"
  ],
  "readme": "README.md",
  "tags": [
    "cloud",
    "virtualization"
  ],
  "description": "Set of common files for the cloud collections",
  "license": [],
  "license_file": "LICENSE",
  "dependencies": {},
  "repository": "https://github.com/ansible-collections/cloud.common",
  "documentation": null,
  "homepage": "https://github.com/ansible-collections/cloud.common",
  "issues": "https://github.com/ansible-collections/cloud.common/issues"
}

@briantist
Copy link
Owner

I see! So the truncation is definitely why it fails on JSON decoding, causing the stacktrace.

https://jfrog.atlassian.net/browse/RTFACT-27074

Nice find on this issue, it says:

depending on type of database used

that part is crazy to me! Are you able to confirm with JFrog if your instance/DB type has a limit and what that limit is?

Also since the properties are actually arrays of strings (IIRC) and not just single strings, I am wondering if the limit is for individual strings, or for the combined length, or something else.


I did search for limits previously and as you found in the docs, they didn't mention any. But this might lead me to have to change the format, and separate some of the info out into separate properties, maybe as an optional configuration. So maybe we'll turn this into a feature request and I'll look at implementing a different method for storing the info.

@mamercad
Copy link
Contributor Author

Are you able to confirm with JFrog if your instance/DB type has a limit and what that limit is?

I'll ask the internal team ...

@mamercad
Copy link
Contributor Author

mamercad commented Feb 1, 2023

But this might lead me to have to change the format

Yeah, what I'm wondering is (it's in the draft PR comments), what are the implications of not having a complete collections_info data structure?

I suppose that each field of collections_info could be its own Artifactory property and Galactory could merge them together ... that might not be too bad ... fqcn, namespace, name, and version are already broken out.

@mamercad
Copy link
Contributor Author

mamercad commented Feb 1, 2023

Another thing that I'm still pondering is, how the heck did a truncated collection_info blob even make it in ... I'd think that json.loads would have traceback'd before the load into Artifactory I'd image (I haven't figured this out yet).

@mamercad
Copy link
Contributor Author

mamercad commented Feb 1, 2023

Oh, another thing ... I did check with the internal team regarding my permissions: I only had read and create (they've since granted me update and delete). Also, it turns out there's a "treat 403s as 404s setting to prevent fuzzing" setting (so, the 404s weren't really true, I was actually getting 403s).

@briantist
Copy link
Owner

briantist commented Feb 1, 2023

Another thing that I'm still pondering is, how the heck did a truncated collection_info blob even make it in ... I'd think that json.loads would have traceback'd before the load into Artifactory I'd image (I haven't figured this out yet).

My guess is that it's artifactory that truncates it on write. The load that triggers the failure I think comes from galactory reading it back.

But this might lead me to have to change the format

Yeah, what I'm wondering is (it's in the draft PR comments), what are the implications of not having a complete collections_info data structure?

I suppose that each field of collections_info could be its own Artifactory property and Galactory could merge them together ... that might not be too bad ... fqcn, namespace, name, and version are already broken out.

Each field could be separate; those other separate ones were the only ones originally. I opted to go (what I thought was) a simpler route by sticking the whole collection_info into its own property.

The way it works right now, only collection_info is needed for galactory (and only sub of the sub-fields are needed, as you've found). The others I left as a way of being able to do easy searches within Artifactory, so folks could implement their own cleanups or other operations outside galactory.

So we could separate out more fields from collection_info, we could also go a route similar to what you have in #54 and trim the structure down to only what's really needed for the software. Will give it some more thought.

Knowing what the real limits are in Artifactory (or your configuration at least, for a single data point), will help me determine which direction to go.

@briantist
Copy link
Owner

Hi @mamercad ! It looks like we might've found a root cause for this in #58 , with a promising fix in #59 .
If you have some time, could you give that fix a try?

@briantist briantist added the bug Something isn't working label Mar 11, 2023
@mamercad
Copy link
Contributor Author

mamercad commented Mar 11, 2023

Hi @mamercad ! It looks like we might've found a root cause for this in #58 , with a promising fix in #59 . If you have some time, could you give that fix a try?

Oh, most excellent -- I should have some cycles next week to give this a shot; I'll report back.

@briantist
Copy link
Owner

@mamercad @jcox10 a fix for this in the artifactory library has been merged, waiting on a release, but I think we'll finally be able to get some closure on these issues:

Once there's a release, I will ensure galactory uses it properly (the behavior is opt-in for now), and probably set the project's minimum version of the library to this new one, stay tuned!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants