Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

global: add Keycloak contrib #217

Merged
merged 11 commits into from
Jan 15, 2021

Conversation

max-moser
Copy link
Contributor

No description provided.

@fenekku fenekku moved this from Triage to In Review in InvenioRDM September Board Sep 1, 2020
@fenekku
Copy link
Contributor

fenekku commented Sep 1, 2020

Hey @max-moser did anyone say they were going to follow-up on this PR? I am not familiar enough (yet!) to give it a fair review, but I think it would be super valuable. I will ping @ntarocco to see if he can (or knows who should) take a look.

Thanks!

Copy link
Contributor

@ntarocco ntarocco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, I am really sorry if it took very long time to review it. I hope it was not really blocking you.

Looks very good to me! Thanks a lot for this contribution, it is great! 👏
This can also replace the cern_openid contrib. at some point, we will just need to "hook in" a way to perform some other actions after login and logout.

I have added a few minor comments, mostly cosmetic if you want to have a look.
If you then need to use this in a future release of invenio-oauthclient, we can probably backport it to a previous version too, let us know.

invenio_oauthclient/contrib/keycloak/__init__.py Outdated Show resolved Hide resolved
invenio_oauthclient/contrib/keycloak/handlers.py Outdated Show resolved Hide resolved
invenio_oauthclient/contrib/keycloak/handlers.py Outdated Show resolved Hide resolved
invenio_oauthclient/contrib/keycloak/helpers.py Outdated Show resolved Hide resolved
invenio_oauthclient/contrib/keycloak/helpers.py Outdated Show resolved Hide resolved
invenio_oauthclient/contrib/keycloak/helpers.py Outdated Show resolved Hide resolved
@max-moser
Copy link
Contributor Author

Hey @ntarocco, thanks for the kind words and feedback! 🙂

I'll incorporate your feedback into the PR and refactor it a bit after the holidays.
And no worries, we have workarounds with own modules in place for the time being 🤓

@lnielsen lnielsen added this to Backlog in InvenioRDM January Board via automation Jan 7, 2021
@lnielsen lnielsen moved this from Triage to Pending merge in InvenioRDM January Board Jan 11, 2021
@ppanero
Copy link
Member

ppanero commented Jan 12, 2021

Tests are:

Passing with Postgresql

-----------------------------------------------------------------------------------------------------
TOTAL                                                                    1401     35    98%

========================= 169 passed, 112 skipped, 218 warnings in 260.06s (0:04:20) =========================

Failing with Mysql

========================================== short test summary info ===========================================
ERROR tests/test_views_rest.py::test_rejected - sqlalchemy.exc.OperationalError: (psycopg2.OperationalError...
============================================== 1 error in 2.06s ==============================================

Stacktrace:

>       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E       sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
E       	Is the server running on host "localhost" (::1) and accepting
E       	TCP/IP connections on port 5432?
E       could not connect to server: Connection refused
E       	Is the server running on host "localhost" (127.0.0.1) and accepting
E       	TCP/IP connections on port 5432?
E
E       (Background on this error at: http://sqlalche.me/e/13/e3q8)

../../../.virtualenvs/invenio-oauthclient/lib/python3.7/site-packages/psycopg2/__init__.py:127: OperationalError

Failing test run alone
Passes...

platform darwin -- Python 3.7.7, pytest-6.0.0, py-1.10.0, pluggy-0.13.1
rootdir: /Users/ppanero/Workspace/inveniosw/invenio-oauthclient, configfile: pytest.ini
plugins: invenio-1.4.1, flask-1.1.0, cov-2.10.1, isort-1.2.0, pydocstyle-2.2.0, pycodestyle-2.2.0
collected 1 item

tests/test_views_rest.py .                                                                             [100%]

============================================== warnings summary ==============================================

@max-moser
Copy link
Contributor Author

The failing tests seem to be due to an accumulation of open connections - when I update the call to pytest in run-tests.sh to ignore a contribution (e.g. python -m pytest --ignore tests/test_contrib_cern.py), the tests pass:

(invenio-oauthclient) maximus@Charlie ~/git/invenio-oauthclient [feature/keycloak*] $ DB=mysql MYSQL_VERSION=MYSQL_8_LATEST SQLALCHEMY_DATABASE_URI="mysql+pymysql://invenio:invenio@localhost:3306/invenio" ./run-tests.sh
lists of files in version control and sdist match
Creating network "docker_services_cli_default" with the default driver
Creating docker_services_cli_mysql_1 ... done
================================================================================== test session starts ===================================================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/maximus/git/invenio-oauthclient, configfile: pytest.ini, testpaths: docs, tests, invenio_oauthclient
plugins: flask-1.1.0, isort-1.2.0, cov-2.10.1, invenio-1.4.1, pycodestyle-2.2.0, pydocstyle-2.2.0
collected 295 items

docs/conf.py ss                                                                                                                                                                    [  0%]
tests/conftest.py sss                                                                                                                                                              [  1%]
tests/helpers.py sss                                                                                                                                                               [  2%]
tests/test_admin.py ss.                                                                                                                                                            [  3%]
tests/test_app.py ss.......                                                                                                                                                        [  6%]
tests/test_base_handlers.py ss..                                                                                                                                                   [  8%]
tests/test_contrib_cern_openid.py ss......                                                                                                                                         [ 10%]
tests/test_contrib_cern_openid_rest.py ss.......                                                                                                                                   [ 13%]
tests/test_contrib_cern_rest.py ss........                                                                                                                                         [ 17%]
tests/test_contrib_github.py ss.......                                                                                                                                             [ 20%]
tests/test_contrib_github_rest.py ss.......                                                                                                                                        [ 23%]
tests/test_contrib_globus.py ss.......                                                                                                                                             [ 26%]
tests/test_contrib_keycloak.py ss..........                                                                                                                                        [ 30%]
tests/test_contrib_orcid.py ss......                                                                                                                                               [ 33%]
tests/test_contrib_orcid_rest.py ss......                                                                                                                                          [ 35%]
tests/test_examples_app.py ss........                                                                                                                                              [ 39%]
tests/test_handlers_rest.py ss................................                                                                                                                     [ 50%]
tests/test_handlers_ui.py ss......                                                                                                                                                 [ 53%]
tests/test_models.py ss....                                                                                                                                                        [ 55%]
tests/test_utils.py ss............                                                                                                                                                 [ 60%]
tests/test_views.py ss.........                                                                                                                                                    [ 64%]
tests/test_views_rest.py ss........                                                                                                                                                [ 67%]
invenio_oauthclient/__init__.py sss                                                                                                                                                [ 68%]
invenio_oauthclient/_compat.py sss                                                                                                                                                 [ 69%]
invenio_oauthclient/admin.py sss                                                                                                                                                   [ 70%]
invenio_oauthclient/config.py sss                                                                                                                                                  [ 71%]
invenio_oauthclient/errors.py sss                                                                                                                                                  [ 72%]
invenio_oauthclient/ext.py sss                                                                                                                                                     [ 73%]
invenio_oauthclient/models.py sss                                                                                                                                                  [ 74%]
invenio_oauthclient/proxies.py sss                                                                                                                                                 [ 75%]
invenio_oauthclient/signals.py sss                                                                                                                                                 [ 76%]
invenio_oauthclient/utils.py sss                                                                                                                                                   [ 77%]
invenio_oauthclient/version.py sss                                                                                                                                                 [ 78%]
invenio_oauthclient/alembic/44ab9963e8cf_create_oauthclient_branch.py sss                                                                                                          [ 79%]
invenio_oauthclient/alembic/97bbc733896c_create_oauthclient_tables.py sss                                                                                                          [ 80%]
invenio_oauthclient/alembic/bff1f190b9bd_add_timestamp_oauthclient.py sss                                                                                                          [ 81%]
invenio_oauthclient/contrib/__init__.py sss                                                                                                                                        [ 82%]
invenio_oauthclient/contrib/cern.py sss                                                                                                                                            [ 83%]
invenio_oauthclient/contrib/cern_openid.py sss                                                                                                                                     [ 84%]
invenio_oauthclient/contrib/github.py sss                                                                                                                                          [ 85%]
invenio_oauthclient/contrib/globus.py sss                                                                                                                                          [ 86%]
invenio_oauthclient/contrib/orcid.py sss                                                                                                                                           [ 87%]
invenio_oauthclient/contrib/keycloak/__init__.py sss                                                                                                                               [ 88%]
invenio_oauthclient/contrib/keycloak/handlers.py sss                                                                                                                               [ 89%]
invenio_oauthclient/contrib/keycloak/helpers.py sss                                                                                                                                [ 90%]
invenio_oauthclient/contrib/keycloak/settings.py sss                                                                                                                               [ 91%]
invenio_oauthclient/handlers/__init__.py sss                                                                                                                                       [ 92%]
invenio_oauthclient/handlers/base.py sss                                                                                                                                           [ 93%]
invenio_oauthclient/handlers/rest.py sss                                                                                                                                           [ 94%]
invenio_oauthclient/handlers/ui.py sss                                                                                                                                             [ 95%]
invenio_oauthclient/handlers/utils.py sss                                                                                                                                          [ 96%]
invenio_oauthclient/views/__init__.py sss                                                                                                                                          [ 97%]
invenio_oauthclient/views/client.py sss                                                                                                                                            [ 98%]
invenio_oauthclient/views/settings.py sss                                                                                                                                          [100%]

==================================================================================== warnings summary ====================================================================================
../../.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/github3/session.py:3
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/github3/session.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Callable

../../.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/github3/structs.py:11
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/github3/structs.py:11: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    class GitHubIterator(models.GitHubCore, collections.Iterator):

tests/test_admin.py: 2 warnings
tests/test_app.py: 1 warning
tests/test_base_handlers.py: 2 warnings
tests/test_contrib_cern_openid.py: 6 warnings
tests/test_contrib_cern_openid_rest.py: 7 warnings
tests/test_contrib_cern_rest.py: 8 warnings
tests/test_contrib_github.py: 7 warnings
tests/test_contrib_github_rest.py: 7 warnings
tests/test_contrib_globus.py: 7 warnings
tests/test_contrib_keycloak.py: 8 warnings
tests/test_contrib_orcid.py: 6 warnings
tests/test_contrib_orcid_rest.py: 6 warnings
tests/test_handlers_rest.py: 40 warnings
tests/test_handlers_ui.py: 6 warnings
tests/test_models.py: 4 warnings
tests/test_utils.py: 12 warnings
tests/test_views.py: 9 warnings
tests/test_views_rest.py: 8 warnings
  /home/maximus/git/invenio-oauthclient/invenio_oauthclient/ext.py:134: DeprecationWarning: CERN Remote app is deprecated, use CERN OpenID instead.
    DeprecationWarning

tests/test_app.py::test_db
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:834: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
    'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '

tests/test_app.py::test_alembic
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/invenio_db/alembic/35c1075e6360_force_naming_convention.py:82: UserWarning: Update accounts_user CHECK None manually
    'manually'.format(c))

tests/test_contrib_cern_openid.py: 5 warnings
tests/test_contrib_cern_openid_rest.py: 6 warnings
tests/test_contrib_cern_rest.py: 5 warnings
tests/test_contrib_github.py: 6 warnings
tests/test_contrib_github_rest.py: 6 warnings
tests/test_contrib_globus.py: 7 warnings
tests/test_contrib_keycloak.py: 7 warnings
tests/test_contrib_orcid.py: 6 warnings
tests/test_contrib_orcid_rest.py: 6 warnings
tests/test_views.py: 8 warnings
tests/test_views_rest.py: 7 warnings
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/flask/sessions.py:211: UserWarning: "localhost" is not a valid cookie domain, it must contain a ".". Add an entry to your hosts file, for example "localhost.localdomain", and use that instead.
    ' "{rv}.localdomain", and use that instead.'.format(rv=rv)

tests/test_utils.py::test_rebuilding_access_tokens
  /home/maximus/.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/invenio_db/utils.py:29: SADeprecationWarning: The Inspector.get_primary_keys() method is deprecated and will be removed in a future release.  Please refer to the Inspector.get_pk_constraint() method.
    primary_key_names = inspector.get_primary_keys(model.__tablename__)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

----------- coverage: platform linux, python 3.7.9-final-0 -----------
Name                                                                    Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------------------------
invenio_oauthclient/__init__.py                                             6      0   100%
invenio_oauthclient/_compat.py                                             16      2    88%   17-18
invenio_oauthclient/admin.py                                               34      0   100%
invenio_oauthclient/alembic/44ab9963e8cf_create_oauthclient_branch.py      11      0   100%
invenio_oauthclient/alembic/97bbc733896c_create_oauthclient_tables.py      24      0   100%
invenio_oauthclient/alembic/bff1f190b9bd_add_timestamp_oauthclient.py      29      0   100%
invenio_oauthclient/config.py                                              10      0   100%
invenio_oauthclient/contrib/__init__.py                                     1      0   100%
invenio_oauthclient/contrib/cern.py                                       164     14    91%   269-276, 422-427, 467-468
invenio_oauthclient/contrib/cern_openid.py                                148      1    99%   228
invenio_oauthclient/contrib/github.py                                      60      0   100%
invenio_oauthclient/contrib/globus.py                                      60      3    95%   260-263
invenio_oauthclient/contrib/keycloak/__init__.py                            4      0   100%
invenio_oauthclient/contrib/keycloak/handlers.py                           40      4    90%   126-129
invenio_oauthclient/contrib/keycloak/helpers.py                            37      4    89%   56, 71, 80, 92
invenio_oauthclient/contrib/keycloak/settings.py                           34      0   100%
invenio_oauthclient/contrib/orcid.py                                       48      0   100%
invenio_oauthclient/errors.py                                              30      0   100%
invenio_oauthclient/ext.py                                                 71      0   100%
invenio_oauthclient/handlers/__init__.py                                   11      0   100%
invenio_oauthclient/handlers/base.py                                       90      6    93%   116, 178, 189, 202, 207, 216
invenio_oauthclient/handlers/rest.py                                       96      4    96%   131, 138, 261, 275
invenio_oauthclient/handlers/ui.py                                         71      4    94%   57, 59, 152, 169
invenio_oauthclient/handlers/utils.py                                      78      0   100%
invenio_oauthclient/models.py                                              73      0   100%
invenio_oauthclient/proxies.py                                              4      0   100%
invenio_oauthclient/signals.py                                              7      0   100%
invenio_oauthclient/utils.py                                              110      0   100%
invenio_oauthclient/version.py                                              3      0   100%
invenio_oauthclient/views/__init__.py                                       5      0   100%
invenio_oauthclient/views/client.py                                       109      9    92%   144, 150-159, 168, 182-193
invenio_oauthclient/views/settings.py                                      33      0   100%
-----------------------------------------------------------------------------------------------------
TOTAL                                                                    1517     51    97%

=============================================================== 153 passed, 142 skipped, 220 warnings in 190.27s (0:03:10) ===============================================================
Stopping docker_services_cli_mysql_1 ... done
Removing docker_services_cli_mysql_1 ... done
Removing network docker_services_cli_default
(invenio-oauthclient) maximus@Charlie ~/git/invenio-oauthclient [feature/keycloak*] $                                                                                          [16:57:31]

@max-moser
Copy link
Contributor Author

With all tests enabled (respectively, none ignored), one of the tests fails:

(invenio-oauthclient) maximus@Charlie ~/git/invenio-oauthclient [feature/keycloak*] $ DB=mysql MYSQL_VERSION=MYSQL_8_LATEST SQLALCHEMY_DATABASE_URI="mysql+pymysql://invenio:invenio@localhost:3306/invenio" ./run-tests.sh
lists of files in version control and sdist match
Creating network "docker_services_cli_default" with the default driver
Creating docker_services_cli_mysql_1 ... done
================================================================================== test session starts ===================================================================================
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/maximus/git/invenio-oauthclient, configfile: pytest.ini, testpaths: docs, tests, invenio_oauthclient
plugins: flask-1.1.0, isort-1.2.0, cov-2.10.1, invenio-1.4.1, pycodestyle-2.2.0, pydocstyle-2.2.0
collected 305 items

docs/conf.py ss                                                                                                                                                                    [  0%]
tests/conftest.py sss                                                                                                                                                              [  1%]
tests/helpers.py sss                                                                                                                                                               [  2%]
tests/test_admin.py ss.                                                                                                                                                            [  3%]
tests/test_app.py ss.......                                                                                                                                                        [  6%]
tests/test_base_handlers.py ss..                                                                                                                                                   [  7%]
tests/test_contrib_cern.py ss........                                                                                                                                              [ 11%]
tests/test_contrib_cern_openid.py ss......                                                                                                                                         [ 13%]
tests/test_contrib_cern_openid_rest.py ss.......                                                                                                                                   [ 16%]
tests/test_contrib_cern_rest.py ss........                                                                                                                                         [ 20%]
tests/test_contrib_github.py ss.......                                                                                                                                             [ 22%]
tests/test_contrib_github_rest.py ss.......                                                                                                                                        [ 25%]
tests/test_contrib_globus.py ss.......                                                                                                                                             [ 28%]
tests/test_contrib_keycloak.py ss..........                                                                                                                                        [ 32%]
tests/test_contrib_orcid.py ss......                                                                                                                                               [ 35%]
tests/test_contrib_orcid_rest.py ss......                                                                                                                                          [ 38%]
tests/test_examples_app.py ss........                                                                                                                                              [ 41%]
tests/test_handlers_rest.py ss................................                                                                                                                     [ 52%]
tests/test_handlers_ui.py ss......                                                                                                                                                 [ 55%]
tests/test_models.py ss....                                                                                                                                                        [ 57%]
tests/test_utils.py ss............                                                                                                                                                 [ 61%]
tests/test_views.py ss.........                                                                                                                                                    [ 65%]
tests/test_views_rest.py ss........E                                                                                                                                               [ 68%]
invenio_oauthclient/__init__.py sss                                                                                                                                                [ 69%]
invenio_oauthclient/_compat.py sss                                                                                                                                                 [ 70%]
invenio_oauthclient/admin.py sss                                                                                                                                                   [ 71%]
invenio_oauthclient/config.py sss                                                                                                                                                  [ 72%]
invenio_oauthclient/errors.py sss                                                                                                                                                  [ 73%]
invenio_oauthclient/ext.py sss                                                                                                                                                     [ 74%]
invenio_oauthclient/models.py sss                                                                                                                                                  [ 75%]
invenio_oauthclient/proxies.py sss                                                                                                                                                 [ 76%]
invenio_oauthclient/signals.py sss                                                                                                                                                 [ 77%]
invenio_oauthclient/utils.py sss                                                                                                                                                   [ 78%]
invenio_oauthclient/version.py sss                                                                                                                                                 [ 79%]
invenio_oauthclient/alembic/44ab9963e8cf_create_oauthclient_branch.py sss                                                                                                          [ 80%]
invenio_oauthclient/alembic/97bbc733896c_create_oauthclient_tables.py sss                                                                                                          [ 81%]
invenio_oauthclient/alembic/bff1f190b9bd_add_timestamp_oauthclient.py sss                                                                                                          [ 82%]
invenio_oauthclient/contrib/__init__.py sss                                                                                                                                        [ 83%]
invenio_oauthclient/contrib/cern.py sss                                                                                                                                            [ 84%]
invenio_oauthclient/contrib/cern_openid.py sss                                                                                                                                     [ 85%]
invenio_oauthclient/contrib/github.py sss                                                                                                                                          [ 86%]
invenio_oauthclient/contrib/globus.py sss                                                                                                                                          [ 87%]
invenio_oauthclient/contrib/orcid.py sss                                                                                                                                           [ 88%]
invenio_oauthclient/contrib/keycloak/__init__.py sss                                                                                                                               [ 89%]
invenio_oauthclient/contrib/keycloak/handlers.py sss                                                                                                                               [ 90%]
invenio_oauthclient/contrib/keycloak/helpers.py sss                                                                                                                                [ 91%]
invenio_oauthclient/contrib/keycloak/settings.py sss                                                                                                                               [ 92%]
invenio_oauthclient/handlers/__init__.py sss                                                                                                                                       [ 93%]
invenio_oauthclient/handlers/base.py sss                                                                                                                                           [ 94%]
invenio_oauthclient/handlers/rest.py sss                                                                                                                                           [ 95%]
invenio_oauthclient/handlers/ui.py sss                                                                                                                                             [ 96%]
invenio_oauthclient/handlers/utils.py sss                                                                                                                                          [ 97%]
invenio_oauthclient/views/__init__.py sss                                                                                                                                          [ 98%]
invenio_oauthclient/views/client.py sss                                                                                                                                            [ 99%]
invenio_oauthclient/views/settings.py sss                                                                                                                                          [100%]

========================================================================================= ERRORS =========================================================================================
___________________________________________________________________________ ERROR at teardown of test_rejected ___________________________________________________________________________

[...]

data = b'\xff\x10\x04#08004Too many connections'

    def raise_mysql_exception(data):
        errno = struct.unpack("<h", data[1:3])[0]
        errval = data[9:].decode("utf-8", "replace")
        errorclass = error_map.get(errno)
        if errorclass is None:
            errorclass = InternalError if errno < 1000 else OperationalError
>       raise errorclass(errno, errval)
E       sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1040, 'Too many connections')
E       (Background on this error at: http://sqlalche.me/e/13/e3q8)

../../.local/share/virtualenvs/invenio-oauthclient-RJ1IXkwX/lib/python3.7/site-packages/pymysql/err.py:143: OperationalError

[...]

================================================================================ short test summary info =================================================================================
ERROR tests/test_views_rest.py::test_rejected - sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1040, 'Too many connections')
========================================================== 161 passed, 144 skipped, 233 warnings, 1 error in 194.49s (0:03:14) ===========================================================

@ppanero ppanero merged commit 93fa74c into inveniosoftware:master Jan 15, 2021
InvenioRDM January Board automation moved this from Pending merge to Done Jan 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

4 participants