Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add galaxy v3 support, refactor, other fixes #104

Merged
merged 48 commits into from
Oct 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
a247fa9
add new models and a few basic tests
briantist Oct 6, 2023
e5dc9a8
move some fixtures
briantist Oct 6, 2023
3547e49
update discover_collections to use model
briantist Oct 6, 2023
57bcb62
convert v2 collections view to use model, add extra pagination fields…
briantist Oct 6, 2023
26c472f
convert v2 collection view to use model, fix collection URLs
briantist Oct 6, 2023
01cc382
convert v2 versions view to use model, add extra pagination fields
briantist Oct 6, 2023
21341fb
convert v2 version view to use model, fix collection URLs
briantist Oct 6, 2023
3ca7a9e
update publish task url generation
briantist Oct 6, 2023
a787ca1
add missing import, remove unused imports
briantist Oct 6, 2023
97bb39b
remove unused functions and imports
briantist Oct 6, 2023
9157761
upstream - skip additional upstream field
briantist Oct 6, 2023
9f03348
upstream - remove client request pagination, force upstream paginatio…
briantist Oct 6, 2023
c5206a4
stop passing scheme to discover_collections
briantist Oct 6, 2023
247ce3d
support prerelease versions in fast detection
briantist Oct 6, 2023
547467a
add equality test to col/ver tests
briantist Oct 6, 2023
4f5eeb2
add v3 structure
briantist Oct 6, 2023
c32ca45
separate output
briantist Oct 6, 2023
813b5e1
fix upstream pagination addition
briantist Oct 6, 2023
4e2964f
fix variable overwrite
briantist Oct 6, 2023
59e23c6
better download URL for upstream results
briantist Oct 6, 2023
ece2140
remove extra comma so its not an interable
briantist Oct 6, 2023
e827066
comma nit
briantist Oct 6, 2023
f877ec5
support original URL passing for downloads
briantist Oct 6, 2023
54c880f
use relative blueprint form for url_for
briantist Oct 6, 2023
1b51084
add and enable v3 endpoints
briantist Oct 6, 2023
f619e23
remove another unnecessary jsonify
briantist Oct 6, 2023
17fcbd2
set up publishing/imports for v3
briantist Oct 6, 2023
bf20590
nit
briantist Oct 7, 2023
77ec5b5
fix upstream download_url rewriting
briantist Oct 7, 2023
0d6cf69
add changelog fragment
briantist Oct 7, 2023
0fe7e16
add known issues
briantist Oct 7, 2023
897145a
make API versions configurable
briantist Oct 7, 2023
c00ff76
update API registratin info logs
briantist Oct 7, 2023
f701fc2
allow unconfigred API_VERSION
briantist Oct 8, 2023
881d539
make app fixture configurable
briantist Oct 8, 2023
bac1e33
update api and collection import tests
briantist Oct 8, 2023
1e124e8
add CollectionData.from_artifactory_path test
briantist Oct 8, 2023
83d786d
xfail new test on >= 3.10
briantist Oct 8, 2023
b710d08
add some tests for CollectionGroup
briantist Oct 8, 2023
9af2a32
add CollectionGroup.from_collection test
briantist Oct 8, 2023
f49276f
more model tweaking and tests
briantist Oct 8, 2023
548fc0f
more CollectionData tests
briantist Oct 8, 2023
2769567
centralize fixture
briantist Oct 8, 2023
8355632
actually add the file
briantist Oct 8, 2023
9734879
add collection_data_factory fixture, use it
briantist Oct 8, 2023
f56a6f3
add some simplistic tests for CollectionCollection
briantist Oct 8, 2023
f4d9cbe
nit: not f-strings
briantist Oct 8, 2023
96541ae
nit
briantist Oct 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# galactory
galactory is An Ansible Galaxy proxy for Artifactory.

Using an Artifactory Generic repository as its backend, galactory implements a limited subset of the Galaxy v2 API to allow for installing and publishing collections.
Using an Artifactory Generic repository as its backend, galactory implements a limited subset of the Galaxy API (v2 & v3) to allow for installing and publishing collections.

It can also be set up to transparently proxy an upstream Galaxy server, storing the pulled artifacts in Artifactory, to be served as local artifacts from then on. This helps avoid throttling errors on busy CI systems, and allows for internal/private collections to declare dependencies on upstream collections (dependencies will only be installed from the same Galaxy server where a collection was installed from).

Expand Down Expand Up @@ -45,6 +45,7 @@ usage: python -m galactory [-h] [-c CONFIG] [--listen-addr LISTEN_ADDR]
[--cache-read CACHE_READ] [--cache-write CACHE_WRITE]
[--use-property-fallback]
[--health-check-custom-text HEALTH_CHECK_CUSTOM_TEXT]
[--api-version {v2,v3}]

galactory is a partial Ansible Galaxy proxy that uploads and downloads collections, using an
Artifactory generic repository as its backend.
Expand Down Expand Up @@ -133,6 +134,10 @@ optional arguments:
--health-check-custom-text HEALTH_CHECK_CUSTOM_TEXT
Sets custom_text field for health check endpoint responses.
[env var: GALACTORY_HEALTH_CHECK_CUSTOM_TEXT]
--api-version {v2,v3}
The API versions to serve. Can be set to limit functionality to specific versions only.
Defaults to all supported versions.
[env var: GALACTORY_API_VERSION]

Args that start with '--' (eg. --listen-addr) can also be set in a config file
(/etc/galactory.d/*.conf or ~/.galactory/*.conf or specified via -c). Config file syntax allows:
Expand Down
19 changes: 19 additions & 0 deletions changelogs/fragments/104-v3-refactor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
major_changes:
- Galactory now supports v3 of the Galaxy API, both as a server, and as a client for upstream proxying (https://github.com/briantist/galactory/issues/23, https://github.com/briantist/galactory/pull/104).

minor_changes:
- Requests that are proxied to an upstream now override the query string to always request 100 results. This is a slight, partial mitigation for our current lack of true pagination suppport (https://github.com/briantist/galactory/issues/99, https://github.com/briantist/galactory/pull/104).
- The ``download`` endpoint now supports a query string parameter ``galactory_upstream_url``. When this parameter is set, and the requested artifact does not already exist in artifactory, this exact URL will be used as the upstream location for this file. If the parameter is not set, it uses the previous proxying behavior for downloads, which is to append the route to the configured upstream to determine its URL. API responses from galactory that contain a ``download_url`` field will automatically generate proper URLs with this parameter set, and it does not require additional configuration or user concern (https://github.com/briantist/galactory/pull/104).
- Added the ``API_VERSION`` multi-valued config option to control which version(s) of the galaxy API are supported for the running instance (https://github.com/briantist/galactory/pull/104).

bugfixes:
- The ``href`` field of several API responses was incorrect, pointing back at the URL requested instead of pointing at the collection endpoint (https://github.com/briantist/galactory/issues/103).

trivial:
- Fast detection did not support detecting prerelease collections (https://github.com/briantist/galactory/issues/100).
- A lot of internal methods have been refactored into custom classes to represent the data, in order to help separate the models from the API version and response formats (https://github.com/briantist/galactory/pull/104).

known_issues:
- Galactory does not support proper paginated responses or proxying. This has always been the case and is not new to this release, but this bug is now tracked (https://github.com/briantist/galactory/issues/99).
- Upstream proxying does not support translation of requests between API versions. While galactory can serve both v2 and v3 simultaneously, an individual request will be proxied to the upstream as is, so the configured upstream must support the version of the request (https://github.com/briantist/galactory/pull/104).
6 changes: 4 additions & 2 deletions galactory/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from .utilities import DateTimeIsoFormatJSONProvider

from .api import bp as api
from .api import create_blueprint as create_api_blueprint
from .download import bp as download
from .health import bp as health
from .root import bp as root
Expand All @@ -23,7 +23,7 @@
app.config.update(**config)
app.register_blueprint(health)
app.register_blueprint(root)
app.register_blueprint(api)
app.register_blueprint(create_api_blueprint(app))
app.register_blueprint(download)

@app.before_request
Expand Down Expand Up @@ -99,6 +99,7 @@
parser.add_argument('--cache-write', action=_StrBool, default=True, env_var='GALACTORY_CACHE_WRITE', help='Populate the upstream cache in Artifactory. Should be false when no auth is provided or the auth has no permission to write.')
parser.add_argument('--use-property-fallback', action='store_true', env_var='GALACTORY_USE_PROPERTY_FALLBACK', help='Set properties of an uploaded collection in a separate request after publshinng. Requires a Pro license of Artifactory. This feature is a workaround for an Artifactory proxy configuration error and may be removed in a future version.')
parser.add_argument('--health-check-custom-text', type=str, default='', env_var='GALACTORY_HEALTH_CHECK_CUSTOM_TEXT', help='Sets custom_text field for health check endpoint responses.')
parser.add_argument('--api-version', action='append', choices=['v2', 'v3'], env_var='GALACTORY_API_VERSION', help='The API versions to serve. Can be set to limit functionality to specific versions only. Defaults to all supported versions.')

Check warning on line 102 in galactory/__init__.py

View check run for this annotation

Codecov / codecov/patch

galactory/__init__.py#L102

Added line #L102 was not covered by tests

if parse_known_only:
args, _ = parser.parse_known_args()
Expand Down Expand Up @@ -175,6 +176,7 @@
CACHE_WRITE=args.cache_write,
USE_PROPERTY_FALLBACK=args.use_property_fallback,
HEALTH_CHECK_CUSTOM_TEXT=args.health_check_custom_text,
API_VERSION=args.api_version,
)

if proxy_fix:
Expand Down
42 changes: 27 additions & 15 deletions galactory/api/__init__.py
Original file line number Diff line number Diff line change
@@ -1,22 +1,34 @@
# -*- coding: utf-8 -*-
# (c) 2022 Brian Scholer (@briantist)

from flask import Blueprint, jsonify
from flask import Blueprint, Flask

from .v2 import bp as v2

API_RESPONSE = {
'available_versions': {
'v2': 'v2/',
},
'current_version': 'v2',
'description': 'GALAXY REST API',
}
def create_blueprint(app: Flask):
api_version = app.config.get('API_VERSION')
response = {
'available_versions': {},
'description': 'GALAXY REST API',
}

bp = Blueprint('api', __name__, url_prefix='/api')
bp.register_blueprint(v2)
bp = Blueprint('api', __name__, url_prefix='/api')

@bp.route('')
@bp.route('/')
def api():
return jsonify(API_RESPONSE)
if api_version is None or 'v3' in api_version:
app.logger.info("Registering Galaxy API v3")
response['available_versions']['v3'] = 'v3/'
from .v3 import bp as v3
bp.register_blueprint(v3)

if api_version is None or 'v2' in api_version:
app.logger.info("Registering Galaxy API v2")
response['available_versions']['v2'] = 'v2/'
response['current_version'] = 'v2' # This field doesn't exist in the v3 output.
from .v2 import bp as v2
bp.register_blueprint(v2)

@bp.route('')
@bp.route('/', endpoint='api')
def api():
return response

return bp
3 changes: 1 addition & 2 deletions galactory/api/v2/collection_imports.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
# (c) 2022 Brian Scholer (@briantist)

from datetime import datetime
from flask import jsonify

from . import bp

Expand All @@ -13,4 +12,4 @@ def import_singleton():
'state': 'completed',
'finished_at': datetime.utcnow(),
}
return jsonify(out)
return out
Loading