Add galaxy v3 support, refactor, other fixes (#104)

* add new models and a few basic tests

* move some fixtures

* update discover_collections to use model

* convert v2 collections view to use model, add extra pagination fields, fix collection URLs

* convert v2 collection view to use model, fix collection URLs

* convert v2 versions view to use model, add extra pagination fields

* convert v2 version view to use model, fix collection URLs

* update publish task url generation

* add missing import, remove unused imports

* remove unused functions and imports

* upstream - skip additional upstream field

* upstream - remove client request pagination, force upstream pagination to 100 (hack)

* stop passing scheme to discover_collections

* support prerelease versions in fast detection

* add equality test to col/ver tests

* add v3 structure

* separate output

* fix upstream pagination addition

* fix variable overwrite

* better download URL for upstream results

* remove extra comma so its not an interable

* comma nit

* support original URL passing for downloads

* use relative blueprint form for url_for

* add and enable v3 endpoints

* remove another unnecessary jsonify

* set up publishing/imports for v3

* nit

* fix upstream download_url rewriting

* add changelog fragment

* add known issues

* make API versions configurable

* update API registratin info logs

* allow unconfigred API_VERSION

* make app fixture configurable

* update api and collection import tests

* add CollectionData.from_artifactory_path test

* xfail new test on >= 3.10

* add some tests for CollectionGroup

* add CollectionGroup.from_collection test

* more model tweaking and tests

* more CollectionData tests

* centralize fixture

* actually add the file

* add collection_data_factory fixture, use it

* add some simplistic tests for CollectionCollection

* nit: not f-strings

* nit

README.md

@@ -3,7 +3,7 @@
 # galactory
 galactory is An Ansible Galaxy proxy for Artifactory.
 
-Using an Artifactory Generic repository as its backend, galactory implements a limited subset of the Galaxy v2 API to allow for installing and publishing collections.
+Using an Artifactory Generic repository as its backend, galactory implements a limited subset of the Galaxy API (v2 & v3) to allow for installing and publishing collections.
 
 It can also be set up to transparently proxy an upstream Galaxy server, storing the pulled artifacts in Artifactory, to be served as local artifacts from then on. This helps avoid throttling errors on busy CI systems, and allows for internal/private collections to declare dependencies on upstream collections (dependencies will only be installed from the same Galaxy server where a collection was installed from).
 
@@ -45,6 +45,7 @@ usage: python -m galactory [-h] [-c CONFIG] [--listen-addr LISTEN_ADDR]
                            [--cache-read CACHE_READ] [--cache-write CACHE_WRITE]
                            [--use-property-fallback]
                            [--health-check-custom-text HEALTH_CHECK_CUSTOM_TEXT]
+                           [--api-version {v2,v3}]
 
 galactory is a partial Ansible Galaxy proxy that uploads and downloads collections, using an
 Artifactory generic repository as its backend.
@@ -133,6 +134,10 @@ optional arguments:
   --health-check-custom-text HEALTH_CHECK_CUSTOM_TEXT
                         Sets custom_text field for health check endpoint responses.
                         [env var: GALACTORY_HEALTH_CHECK_CUSTOM_TEXT]
+  --api-version {v2,v3}
+                        The API versions to serve. Can be set to limit functionality to specific versions only.
+                        Defaults to all supported versions.
+                        [env var: GALACTORY_API_VERSION]
 
 Args that start with '--' (eg. --listen-addr) can also be set in a config file
 (/etc/galactory.d/*.conf or ~/.galactory/*.conf or specified via -c). Config file syntax allows:

changelogs/fragments/104-v3-refactor.yml

@@ -0,0 +1,19 @@
+---
+major_changes:
+  - Galactory now supports v3 of the Galaxy API, both as a server, and as a client for upstream proxying (https://github.com/briantist/galactory/issues/23, https://github.com/briantist/galactory/pull/104).
+
+minor_changes:
+  - Requests that are proxied to an upstream now override the query string to always request 100 results. This is a slight, partial mitigation for our current lack of true pagination suppport (https://github.com/briantist/galactory/issues/99, https://github.com/briantist/galactory/pull/104).
+  - The ``download`` endpoint now supports a query string parameter ``galactory_upstream_url``. When this parameter is set, and the requested artifact does not already exist in artifactory, this exact URL will be used as the upstream location for this file. If the parameter is not set, it uses the previous proxying behavior for downloads, which is to append the route to the configured upstream to determine its URL. API responses from galactory that contain a ``download_url`` field will automatically generate proper URLs with this parameter set, and it does not require additional configuration or user concern (https://github.com/briantist/galactory/pull/104).
+  - Added the ``API_VERSION`` multi-valued config option to control which version(s) of the galaxy API are supported for the running instance (https://github.com/briantist/galactory/pull/104).
+
+bugfixes:
+  - The ``href`` field of several API responses was incorrect, pointing back at the URL requested instead of pointing at the collection endpoint (https://github.com/briantist/galactory/issues/103).
+
+trivial:
+  - Fast detection did not support detecting prerelease collections (https://github.com/briantist/galactory/issues/100).
+  - A lot of internal methods have been refactored into custom classes to represent the data, in order to help separate the models from the API version and response formats (https://github.com/briantist/galactory/pull/104).
+
+known_issues:
+  - Galactory does not support proper paginated responses or proxying. This has always been the case and is not new to this release, but this bug is now tracked (https://github.com/briantist/galactory/issues/99).
+  - Upstream proxying does not support translation of requests between API versions. While galactory can serve both v2 and v3 simultaneously, an individual request will be proxied to the upstream as is, so the configured upstream must support the version of the request (https://github.com/briantist/galactory/pull/104).

galactory/__init__.py

@@ -11,7 +11,7 @@
 
 from .utilities import DateTimeIsoFormatJSONProvider
 
-from .api import bp as api
+from .api import create_blueprint as create_api_blueprint
 from .download import bp as download
 from .health import bp as health
 from .root import bp as root
@@ -23,7 +23,7 @@ def create_app(**config):
     app.config.update(**config)
     app.register_blueprint(health)
     app.register_blueprint(root)
-    app.register_blueprint(api)
+    app.register_blueprint(create_api_blueprint(app))
     app.register_blueprint(download)
 
     @app.before_request
@@ -99,6 +99,7 @@ def create_configured_app(run=False, parse_known_only=True, parse_allow_abbrev=F
     parser.add_argument('--cache-write', action=_StrBool, default=True, env_var='GALACTORY_CACHE_WRITE', help='Populate the upstream cache in Artifactory. Should be false when no auth is provided or the auth has no permission to write.')
     parser.add_argument('--use-property-fallback', action='store_true', env_var='GALACTORY_USE_PROPERTY_FALLBACK', help='Set properties of an uploaded collection in a separate request after publshinng. Requires a Pro license of Artifactory. This feature is a workaround for an Artifactory proxy configuration error and may be removed in a future version.')
     parser.add_argument('--health-check-custom-text', type=str, default='', env_var='GALACTORY_HEALTH_CHECK_CUSTOM_TEXT', help='Sets custom_text field for health check endpoint responses.')
+    parser.add_argument('--api-version', action='append', choices=['v2', 'v3'], env_var='GALACTORY_API_VERSION', help='The API versions to serve. Can be set to limit functionality to specific versions only. Defaults to all supported versions.')
 
     if parse_known_only:
         args, _ = parser.parse_known_args()
@@ -175,6 +176,7 @@ def create_configured_app(run=False, parse_known_only=True, parse_allow_abbrev=F
         CACHE_WRITE=args.cache_write,
         USE_PROPERTY_FALLBACK=args.use_property_fallback,
         HEALTH_CHECK_CUSTOM_TEXT=args.health_check_custom_text,
+        API_VERSION=args.api_version,
     )
 
     if proxy_fix:

galactory/api/__init__.py

@@ -1,22 +1,34 @@
 # -*- coding: utf-8 -*-
 # (c) 2022 Brian Scholer (@briantist)
 
-from flask import Blueprint, jsonify
+from flask import Blueprint, Flask
 
-from .v2 import bp as v2
 
-API_RESPONSE = {
-    'available_versions': {
-        'v2': 'v2/',
-    },
-    'current_version': 'v2',
-    'description': 'GALAXY REST API',
-}
+def create_blueprint(app: Flask):
+    api_version = app.config.get('API_VERSION')
+    response = {
+        'available_versions': {},
+        'description': 'GALAXY REST API',
+    }
 
-bp = Blueprint('api', __name__, url_prefix='/api')
-bp.register_blueprint(v2)
+    bp = Blueprint('api', __name__, url_prefix='/api')
 
-@bp.route('')
-@bp.route('/')
-def api():
-    return jsonify(API_RESPONSE)
+    if api_version is None or 'v3' in api_version:
+        app.logger.info("Registering Galaxy API v3")
+        response['available_versions']['v3'] = 'v3/'
+        from .v3 import bp as v3
+        bp.register_blueprint(v3)
+
+    if api_version is None or 'v2' in api_version:
+        app.logger.info("Registering Galaxy API v2")
+        response['available_versions']['v2'] = 'v2/'
+        response['current_version'] = 'v2' # This field doesn't exist in the v3 output.
+        from .v2 import bp as v2
+        bp.register_blueprint(v2)
+
+    @bp.route('')
+    @bp.route('/', endpoint='api')
+    def api():
+        return response
+
+    return bp

galactory/api/v2/collection_imports.py

@@ -2,7 +2,6 @@
 # (c) 2022 Brian Scholer (@briantist)
 
 from datetime import datetime
-from flask import jsonify
 
 from . import bp
 
@@ -13,4 +12,4 @@ def import_singleton():
         'state': 'completed',
         'finished_at': datetime.utcnow(),
     }
-    return jsonify(out)
+    return out