Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server OGC API: fix url path match #45450

Merged
merged 4 commits into from
Oct 8, 2021

Conversation

elpaso
Copy link
Contributor

@elpaso elpaso commented Oct 7, 2021

Fixes #45439

@elpaso elpaso added Bug Either a bug report, or a bug fix. Let's hope for the latter! Server Related to QGIS server labels Oct 7, 2021
@github-actions github-actions bot added this to the 3.22.0 milestone Oct 7, 2021
@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

CC @dmarteau can you give this a try?

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

@elpaso I'm doing that ATM.

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

Here is the results of same runs as #45439 :

qgis-server_1  | 2021-10-07 08:09:09,913	DEBUG	[63]	Qgis: Server: API TileMapService accepts the URL path '/tms/france_parts.json/' 
qgis-server_1  | 2021-10-07 08:09:09,913	DEBUG	[63]	Qgis: Server: Checking API path /tms/france_parts.json for /(?P<tilemapid>[^/]+) 
qgis-server_1  | 2021-10-07 08:09:09,913	DEBUG	[63]	Qgis: Server: API TileMapService: found handler tilesTileMapInfo
# Here is a check of the parameter in the handler implementation,
# OK This result is expected as the the url path starts with '/tms/'
qgis-server_1  | 2021-10-07 08:09:09,914	ERROR	[63]	Qgis: tilesApi: Tile map 'tms' not found

Tests with query params using: http://localhost:19876/tms/france_parts.json?MAP=france_parts

qgis-server_1  | 2021-10-07 08:14:39,602	DEBUG	[63]	Qgis: Server: API TileMapService accepts the URL path '/tms/france_parts.json'  
qgis-server_1  | 2021-10-07 08:14:39,602	DEBUG	[63]	Qgis: Server: Checking API path /tms/france_parts.json for /tms/(?P<tilemapid>[^/]+) 
qgis-server_1  | 2021-10-07 08:14:39,602	DEBUG	[63]	Qgis: Server: API TileMapService: found handler tilesTileMapInfo
# Here is a check of the parameter in the handler implementation
# OK, no more query params
qgis-server_1  | 2021-10-07 08:14:39,602	ERROR	[63]	Qgis: tilesApi: Tile map 'france_parts.json' not found
 found

The only thing a little confusing at start is that the match is not an exact match but rather finding a substring that match.
So the results may be different, if you use a pattern that try to capture the first part or just matching a word:

  • /(?P<whatever>[^/]+) will return the rootpath (i.e tms in the case above) so one has to use /rootpath/(?P<whatever>[^/]+) to get the expected result.
  • /collections/ -> will match /tms/collections/

But that's fine - since I can still use the exact match for the pattern (I am more acustomed to exact match when using web server framework) and changing this behavior may introduce some regression in existing implementations. May be this should be emphasized in the documention.

Thanks for the fix.

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

@elpaso I think I found another issue when computing href with the QgsServerOgcApiHandler::href method if you prefix the pattern with the apiroot

Example: if I use /apiroot/<pattern> then computed href will have twice the api root component in ther path:

Using /tms/? for the landing page give the following result with href:

href('/france_parts',QgsServerOgcApi.contentTypeToExtension(QgsServerOgcApi.JSON)) returns http://localhost:19876/tms/tms/france_parts.json?MAP=france_parts

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

@dmarteau the root path is designed to be like a namespace (possibly configurable) and it's not part of the handlers API, that's why it's not a regexp pattern.

Considering WFS3 as an example, /wfs3 is the root path and /wfs3 does not appear anywhere in the handlers. In other words: do not prepend the root path to the handler's path, the root path is designed to stay out of the way (and possibly be changed).

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

the root path is designed to be like a namespace (possibly configurable) and it's not part of the handlers API, that's why it's not a regexp pattern.

I understand, but actually if I use a pattern like /(?P<whatever>[^/]+) for a handler, the returned value for whatever is the root path.

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

the root path is designed to be like a namespace (possibly configurable) and it's not part of the handlers API, that's why it's not a regexp pattern.

I understand, but actually if I use a pattern like /(?P<whatever>[^/]+) for a handler, the returned value for whatever is the root path.

Sorry but I find this discussion hard to follow: returned from what? Would you mind filing a separate ticket if this is unrelated from this PR?

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

@elpaso have you some time to spare for a chat ?

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

@elpaso have you some time to spare for a chat ?

Not now, sorry :/

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

this is unrelated from this PR?

This is a direct consequence of this PR (otherwise you just bump into the problem describe in the issue #45439)

I'm trying to rephrase the problem:

The context

  1. I create QgsServerOgcApi with the rootpath /myapi:
  2. I register a handler with the following pattern /(?P<name>[^/]+)/?

Then I expect name to be an entry in the map returned from QgsServerOgcApiHandler::values.

Expected result

If the request is http://my.host.com/myapi/foobar, I expect the value corresponding to name to be foobar

Actual result

The value I get for name is myapi

I can workaround this if I prefix my handler regexp with /myapi which is - from your statement above - not how it is expected to write expression for handlers (i.e beeing not prefixed with the root api).

And if I try to workaround the issue by prefixing the handler expression, I have the side effect that computed href are wrong (as described in #45450 (comment))

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

this is unrelated from this PR?

This is a direct consequence of this PR (otherwise you just bump into the problem describe in the issue #45439)

I'm trying to rephrase the problem:

The context

1. I create  QgsServerOgcApi with the rootpath `/myapi`:

2. I register a handler with the following pattern `/(?P<name>[^/]+)/?`

Then I expect name to be an entry in the map returned from QgsServerOgcApiHandler::values.

Expected result

If the request is http://my.host.com/myapi/foobar, I expect the value corresponding to name to be foobar

Actual result

The value I get for name is myapi

I can workaround this if I prefix my handler regexp with /myapi which is - from your statement above - not how it is expected to write expression for handlers (i.e beeing not prefixed with the root api).

And if I try to workaround the issue by prefixing the handler expression, I have the side effect that computed href are wrong (as described in #45450 (comment))

Much clearer now, thanks. If you want to be even more clearer (and make my life easier) you can provide a failing test case.

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

Much clearer now, thanks. If you want to be even more clearer (and make my life easier) you can provide a failing test case.

Nevermind, got a failing test myself.

@elpaso
Copy link
Contributor Author

elpaso commented Oct 7, 2021

Much clearer now, thanks. If you want to be even more clearer (and make my life easier) you can provide a failing test case.

Nevermind, got a failing test myself.

Wait: this passes for me

    def test_path_capture(self):
        """Test issue GH #45439"""
        project = QgsProject()

        api = QgsServerOgcApi(self.server.serverInterface(),
                              '/services/api4', 'apifour', 'a fourth api', '1.2')

        h4 = Handler4()
        api.registerHandler(h4)

        request = QgsBufferServerRequest(
            'http://localhost:19876/services/api4/tms/france_parts.json?MAP=france_parts')
        response = QgsBufferServerResponse()

        server = QgsServer()
        iface = server.serverInterface()
        iface.serviceRegistry().registerApi(api)

        server.handleRequest(request, response)

        self.assertEqual(h4.params, {'tilemapid': 'france_parts.json'})

        ctx = QgsServerApiContext(api.rootPath(), request, response, project, iface)
        self.assertEqual(h4.href(ctx), 'http://localhost:19876/services/api4/tms/france_parts?MAP=france_parts')

And now I realize that you claim is wrong:

Expected result
If the request is http://my.host.com/myapi/foobar, I expect the value corresponding to name to be foobar

You are missing a trailing slash, the regexp won't match the URL.

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

Here is the result of my failing tests: (I am building a simpler one)

Request is http://localhost:19876/tms/foobar?MAP=france_parts

Case 1: wrong result for tilemapid wrong href

qgis-server_1  | 2021-10-07 16:01:09,603	DEBUG	[63]	Qgis: Server: API TileMapService accepts the URL path '/tms/foobar' 
qgis-server_1  | 2021-10-07 16:01:09,603	DEBUG	[63]	Qgis: Server: Checking API path /tms/foobar for /(?P<tilemapid>(?:(?!\.json)[^/\?])+) 
# Handler path is matching
qgis-server_1  | 2021-10-07 16:01:09,603	DEBUG	[63]	Qgis: Server: API TileMapService: found handler tilesTileMapInfo
qgis-server_1  | 2021-10-07 16:01:09,604	WARNING	[63]	Qgis: tilesApi: href is 'http://localhost:19876/tms/tms/tms.json?MAP=france_parts'
# Got 'tms' as result for 'tilemapid' KO
qgis-server_1  | 2021-10-07 16:01:09,604	ERROR	[63]	Qgis: tilesApi: Tile map 'tms' not found

Case 2 (prefixed): ok for tilemapid, wrong href

qgis-server_1  | 2021-10-07 16:03:37,606	DEBUG	[63]	Qgis: Server: Trying URL path: '/tms/foobar' for '/tms'
qgis-server_1  | 2021-10-07 16:03:37,606	DEBUG	[63]	Qgis: Server: API TileMapService accepts the URL path '/tms/foobar' # Handler path is matching
qgis-server_1  | 2021-10-07 16:03:37,606	DEBUG	[63]	Qgis: Server: Checking API path /tms/foobar for /tms/(?P<tilemapid>(?:(?!\.json)[^/\?])+) 
qgis-server_1  | 2021-10-07 16:03:37,606	DEBUG	[63]	Qgis: Server: API TileMapService: found handler tilesTileMapInfo
# Href is wrong
qgis-server_1  | 2021-10-07 16:03:37,607	WARNING	[63]	Qgis: tilesApi: href is 'http://localhost:19876/tms/tms/foobar/foobar.json?MAP=france_parts'
# Got 'foobar' as result for 'tilemapid' OK
qgis-server_1  | 2021-10-07 16:03:37,607	ERROR	[63]	Qgis: tilesApi: Tile map 'foobar' not found

Note that in both case href is wrong, adding a trailing slash to the url does not change the result.

@dmarteau
Copy link
Contributor

dmarteau commented Oct 7, 2021

Here is may failing test:

""" Failing test for https://github.com/qgis/QGIS/pull/45450

    Must be run using https://github.com/elpaso/QGIS/tree/bugfix-gh45439-server-api-path-match
"""
import os
from qgis.PyQt.QtCore import QRegularExpression
from qgis.core import Qgis, QgsMessageLog, QgsApplication
from qgis.server import (
    QgsServerOgcApi,
    QgsServerOgcApiHandler,
    QgsBufferServerRequest,
    QgsBufferServerResponse,
    QgsServer,
)


class TestHandler(QgsServerOgcApiHandler):
    
    def __init__(self, path: str):
        super().__init__()
        self._path = QRegularExpression(path)
        self._name = 'test'

    def handleRequest(self, context):
        """
        """
        values = self.values(context)
        print("#####> Values",values)
        print("#####> href", self.href(context,"/test", "json"))
        context.response().setStatusCode(200)

    def path(self):
        return self._path

    def linkType(self):
        return QgsServerOgcApi.items

    def operationId(self):
        return f"{self._name}"

    def summary(self):
        return f"Tiles {self._name}"

    def description(self):
        return f"Tiles {self._name}"

    def linkTitle(self):
        return f"Tiles {self._name}"

    def templatePath(self, context):
        # No templates!
        return ''

    def parameters(self, context):
        return []


def test_pr45540() -> None:
    """
    """
    os.environ['QT_QPA_PLATFORM'] = 'offscreen'

    qgis_application = QgsApplication([], False)
    qgis_application.initQgis()

    install_logger_hook()

    server = QgsServer()  
    
    iface = server.serverInterface()
    api     = QgsServerOgcApi(iface, '/myapi','Test PR 45450')    
    handler = TestHandler('/(?P<name>[^/]+)/?')

    api.registerHandler(handler)

    iface.serviceRegistry().registerApi(api)

    request = QgsBufferServerRequest('http://localhost:19876/myapi/foobar?MAP=tests/data/france_parts.qgs')
    response = QgsBufferServerResponse()

    server.handleRequest(request, response)

    qgis_application.exitQgis()


def install_logger_hook() -> None:
    """ Install message log hook
    """
    from qgis.core import Qgis, QgsApplication, QgsMessageLog

    # Add a hook to qgis  message log
    def writelogmessage(message, tag, level):
        arg = f'{level}:{tag}: {message}'
        print(arg, flush=True)

    messageLog = QgsApplication.messageLog()
    messageLog.messageReceived.connect( writelogmessage )

test_pr45540()

Results:

...
0:Server: Server initialized
0:: Adding API Test PR 45450 
Logged warning: Loading a file that was saved with an older version of qgis (saved in 3.10.14-A Coruña, loaded in 3.21.0-Master). Problems may occur.
0:Server: Trying URL path: '/myapi/foobar' for '/myapi'
0:Server: API Test PR 45450 accepts the URL path '/myapi/foobar' 
0:Server: Checking API path /myapi/foobar for /(?P<name>[^/]+)/? 
0:Server: API Test PR 45450: found handler test
#####> Values {'name': 'myapi'}
#####> href http://localhost:19876/myapi/myapi/test.json?MAP=tests/data/france_parts.qgs

@elpaso
Copy link
Contributor Author

elpaso commented Oct 8, 2021

@dmarteau works for me if you use this regexp: handler = TestHandler('/(?P<name>[^/]+)/?$'), I'll see if I can make it more robust and match against the path after removing the api part (/myapi).

@elpaso
Copy link
Contributor Author

elpaso commented Oct 8, 2021

@dmarteau your test now passes.

@elpaso elpaso force-pushed the bugfix-gh45439-server-api-path-match branch from d61df93 to 2b1a1c1 Compare October 8, 2021 09:18
Handle the messy initial / by making sure the
paths matches are evaluated against a sanitized
URL that always starts with a /
@elpaso elpaso merged commit 8498c55 into qgis:master Oct 8, 2021
@elpaso elpaso deleted the bugfix-gh45439-server-api-path-match branch October 8, 2021 13:07
@rldhont
Copy link
Contributor

rldhont commented Oct 8, 2021

@elpaso @dmarteau should you do the backport to queued_ltr_backports ?

@elpaso elpaso added the backport queued_ltr_backports Queued Backports label Oct 8, 2021
@qgis-bot
Copy link
Collaborator

qgis-bot commented Oct 8, 2021

The backport to queued_ltr_backports failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-queued_ltr_backports queued_ltr_backports
# Navigate to the new working tree
cd .worktrees/backport-queued_ltr_backports
# Create a new branch
git switch --create backport-45450-to-queued_ltr_backports
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick 7cd8f0931dfb61bb96f8c0429a0896e7b0ab8434,9df14378142605f282841f175deff4bb9ee75660,2b1a1c198682ce8f8b60338997fcc8e3413a151e,8ba18051e42196e69d7a27da23f36b240ff4c771
# Push it to GitHub
git push --set-upstream origin backport-45450-to-queued_ltr_backports
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-queued_ltr_backports

Then, create a pull request where the base branch is queued_ltr_backports and the compare/head branch is backport-45450-to-queued_ltr_backports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport queued_ltr_backports Queued Backports Bug Either a bug report, or a bug fix. Let's hope for the latter! Server Related to QGIS server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Server][Ogc Api] Inconsistency regarding pattern handling for QgsServerOgcApiHandler
4 participants