Traceback when publishing amazon.aws

[mamercad]

When trying to publish amazon.aws:==5.2.0, I'm seeing:

ERROR:galactory:Exception on /api/v2/collections/amazon/aws/ [GET]
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/api/v2/collections.py", line 47, in collection
    results = _collection_listing(repository, namespace, collection, scheme=scheme)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 162, in _collection_listing
    collections = collected_collections(repo, namespace, collection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 143, in collected_collections
    for c in discover_collections(repo, namespace=namespace, name=name, scheme=scheme):
  File "/venv/lib/python3.11/site-packages/galactory/utilities.py", line 107, in discover_collections
    collection_info = json.loads(props['collection_info'][0])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 482 (char 481)

[mamercad]

I have a hunch ... looks like there's not enough metadata ... I don't see fqcn, namespace, name and more.

[mamercad]

I'm hacking on this a bit locally, manually raised a RuntimeError to dump out the properties and they look ok:

RuntimeError: {'collection_info': '{"namespace": "amazon", "name": "aws", "version": "5.2.0", "authors": ["Ansible (https://github.com/ansible)"], "readme": "README.md", "tags": ["amazon", "aws", "cloud"], "description": null, "license": [], "license_file": "COPYING", "dependencies": {}, "repository": "https://github.com/ansible-collections/amazon.aws", "documentation": "https://ansible-collections.github.io/amazon.aws/branch/stable-5/collections/amazon/aws/index.html", "homepage": "https://github.com/ansible-collections/amazon.aws", "issues": "https://github.com/ansible-collections/amazon.aws/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc"}', 'namespace': 'amazon', 'name': 'aws', 'version': '5.2.0', 'fqcn': 'amazon.aws'}

[briantist]

Hi @mamercad !

This happens during publish? The top of the traceback shows a GET request:

ERROR:galactory:Exception on /api/v2/collections/amazon/aws/ [GET]

And the rest of it shows reads (it's a JSON decoding issue that seems to have raised the exception in the first place.

So I'm wondering if there's a longer trace that might show more information on the sequence of events.

Also could you show me the snippet you added that shows the RuntimeError? Maybe do a git diff and post that so I can see it in context.

From this comment, are you seeing that only collection_info is in Artifactory as a property?

When looking at it in Artifactory, is it complete/valid JSON?

Also wondering if you still have that property fallback option enabled, or set it back to doing uploads in one shot.

[mamercad]

...the "missing properties" might be a red herring; I've been digging all day and what I'm seeing are sporadic Artifactory timeouts.

[mamercad]

When looking at it in Artifactory, is it complete/valid JSON?

Ooh, at least from the UI, it does not seem valid, it appears that it's truncated!

...maybe we're hitting a limit.

[mamercad]

Still researching, here's a bug I found upstream which may be related.

[mamercad]

No mention of property constraints here.

[mamercad]

Yup, it's truncated:

❯ curl -s -X GET -H "X-JFrog-Art-Api: ${ARTIFACTORY_API_KEY}" "https://REDACTED/artifactory/api/storage/ansible-dev-local/amazon-aws-5.2.0.tar.gz?properties" | jq -r '.properties.collection_info[0]'    
{"namespace": "amazon", "name": "aws", "version": "5.2.0", "authors": ["Ansible (https://github.com/ansible)"], "readme": "README.md", "tags": ["amazon", "aws", "cloud"], "description": null, "license": [], "license_file": "COPYING", "dependencies": {}, "repository": "https://github.com/ansible-collections/amazon.aws", "documentation": "https://ansible-collections.github.io/amazon.aws/branch/stable-5/collections/amazon/aws/index.html", "homepage": "https://github.com/ansible-collections/amazon.aws", "issues": "https://github.com/ansible-collections/amazon.aws/issues

Versus:

❯ curl -s -X GET -H "X-JFrog-Art-Api: ${ARTIFACTORY_API_KEY}" "https://REDACTED/artifactory/api/storage/ansible-dev-local/cloud-common-2.1.1.tar.gz?properties" | jq -r '.properties.collection_info[0]' | jq
{
  "namespace": "cloud",
  "name": "common",
  "version": "2.1.1",
  "authors": [
    "Ansible (https://github.com/ansible)"
  ],
  "readme": "README.md",
  "tags": [
    "cloud",
    "virtualization"
  ],
  "description": "Set of common files for the cloud collections",
  "license": [],
  "license_file": "LICENSE",
  "dependencies": {},
  "repository": "https://github.com/ansible-collections/cloud.common",
  "documentation": null,
  "homepage": "https://github.com/ansible-collections/cloud.common",
  "issues": "https://github.com/ansible-collections/cloud.common/issues"
}

[briantist]

I see! So the truncation is definitely why it fails on JSON decoding, causing the stacktrace.

https://jfrog.atlassian.net/browse/RTFACT-27074

Nice find on this issue, it says:

depending on type of database used

that part is crazy to me! Are you able to confirm with JFrog if your instance/DB type has a limit and what that limit is?

Also since the properties are actually arrays of strings (IIRC) and not just single strings, I am wondering if the limit is for individual strings, or for the combined length, or something else.

---

I did search for limits previously and as you found in the docs, they didn't mention any. But this might lead me to have to change the format, and separate some of the info out into separate properties, maybe as an optional configuration. So maybe we'll turn this into a feature request and I'll look at implementing a different method for storing the info.

[mamercad]

Are you able to confirm with JFrog if your instance/DB type has a limit and what that limit is?

I'll ask the internal team ...

[mamercad]

But this might lead me to have to change the format

Yeah, what I'm wondering is (it's in the draft PR comments), what are the implications of not having a complete collections_info data structure?

I suppose that each field of collections_info could be its own Artifactory property and Galactory could merge them together ... that might not be too bad ... fqcn, namespace, name, and version are already broken out.

[mamercad]

Another thing that I'm still pondering is, how the heck did a truncated collection_info blob even make it in ... I'd think that json.loads would have traceback'd before the load into Artifactory I'd image (I haven't figured this out yet).

[mamercad]

Oh, another thing ... I did check with the internal team regarding my permissions: I only had read and create (they've since granted me update and delete). Also, it turns out there's a "treat 403s as 404s setting to prevent fuzzing" setting (so, the 404s weren't really true, I was actually getting 403s).

[briantist]

Another thing that I'm still pondering is, how the heck did a truncated collection_info blob even make it in ... I'd think that json.loads would have traceback'd before the load into Artifactory I'd image (I haven't figured this out yet).

My guess is that it's artifactory that truncates it on write. The load that triggers the failure I think comes from galactory reading it back.

But this might lead me to have to change the format

Yeah, what I'm wondering is (it's in the draft PR comments), what are the implications of not having a complete collections_info data structure?

I suppose that each field of collections_info could be its own Artifactory property and Galactory could merge them together ... that might not be too bad ... fqcn, namespace, name, and version are already broken out.

Each field could be separate; those other separate ones were the only ones originally. I opted to go (what I thought was) a simpler route by sticking the whole collection_info into its own property.

The way it works right now, only collection_info is needed for galactory (and only sub of the sub-fields are needed, as you've found). The others I left as a way of being able to do easy searches within Artifactory, so folks could implement their own cleanups or other operations outside galactory.

So we could separate out more fields from collection_info, we could also go a route similar to what you have in #54 and trim the structure down to only what's really needed for the software. Will give it some more thought.

Knowing what the real limits are in Artifactory (or your configuration at least, for a single data point), will help me determine which direction to go.

[briantist]

Hi @mamercad ! It looks like we might've found a root cause for this in #58 , with a promising fix in #59 .
If you have some time, could you give that fix a try?

[mamercad]

Hi @mamercad ! It looks like we might've found a root cause for this in #58 , with a promising fix in #59 . If you have some time, could you give that fix a try?

Oh, most excellent -- I should have some cycles next week to give this a shot; I'll report back.

[briantist]

@mamercad @jcox10 a fix for this in the artifactory library has been merged, waiting on a release, but I think we'll finally be able to get some closure on these issues:

• URL quoting for matrix parameters devopshq/artifactory#409

Once there's a release, I will ensure galactory uses it properly (the behavior is opt-in for now), and probably set the project's minimum version of the library to this new one, stay tuned!