before_(app_)first_request deprecation and removal in Flask v2.3

[slint]

In Flask v2.2 before_(app_)first_request was deprecated and will be removed in v2.3.

We are currently using this through various modules for registering menus or other special cases:

• before_first_request
• before_app_first_request
• (Optional, but similar) Blueprint.record_once (see before_(app_)first_request deprecation and removal in Flask v2.3 #171 (comment) 👇 )

We should either:

• Pin Flask < 2.3 in this module
• Refactor the code not to use these decorators. This is pretty difficult since the proposed alternative is to "Run setup code when creating the application instead."... This means we have to hook in the invenio-app lifecycle to introduce a similar mechanism.

[jrcastro2]

11 repositories affected by this.

• flask-menu:
  • classy.py
  • __init__.py
• invenio-administration
  • ext.py
  • admin.py
• invenio-github:
  • ext.py
• invenio-records-rest
  • test_custom_endpoints.py
• invenio-deposit
  • ext.py
• invenio-admin
  • ext.py
  • test_invenio_admin.py
  • views.py
• invenio-oauthclient
  • ext.py
• invenio-oauth2server
  • consumer.py
• invenio-accounts
  • settings.py
• invenio-app-rdm
  • views.py
  • ui.py
  • communities_ui/views/ui.py
• invenio-communities
  • ui.py
• flask-security-fork
  • core.py
• invenio-pages
  • views.py

Some usefull resources:

• Discussion about before_first_request with some potential solutions: Use case for @before_first_request pallets/flask#4711

[utnapischtim]

NOTES:

• would that solution also makes @blueprint.record_once decorators obsolete?

[slint]

• would that solution also makes @blueprint.record_once decorators obsolete?

I would say yes (at least from a quick search). The main difference with the before_(app_)first_request usage, is that it doesn't depend on an application context being pushed, but has access to the app via state.app. It still assumes that all extensions have been initialized though.

I think whenever we're using a current_XYZ in these function, we're assuming app context, but it's not necessary actually... If you have the app object, you can usually access any of these current_XYZ objects via app.extensions['<XYZ-module>'].XYZ.

[utnapischtim]

• would that solution also makes @blueprint.record_once decorators obsolete?

I would say yes (at least from a quick search). The main difference with the before_(app_)first_request usage, is that it doesn't depend on an application context being pushed, but has access to the app via state.app. It still assumes that all extensions have been initialized though.

I think whenever we're using a current_XYZ in these function, we're assuming app context, but it's not necessary actually... If you have the app object, you can usually access any of these current_XYZ objects via app.extensions['<XYZ-module>'].XYZ.

Good to know. thank you.

Should i include those examples in the affected list?

Is the proposed method to fix that DeprecationWarning still under discussion or can i go further and apply it?

[slint]

Should i include those examples in the affected list?

Yes, I think they're good candidates for this type of logic refactoring.

Is the proposed method to fix that DeprecationWarning still under discussion or can i go further and apply it?

#172 and inveniosoftware/invenio-app#88 are the "leading" implementations. Personally, I'm still a bit hesitant about the naming (I like more the finalize_init_app naming proposed in flask), but otherwise, I think that's the direction we're going with.

An alternative to adding yet another entrypoint, would be to move the logic in the extensions and introduce an optional finalize_init_app() method, which when exists gets called for all the extensions defined in the original inenio_app.apps and invenio_app.api_apps.

[utnapischtim]

@slint
i need some advice/decision for flask-menu.

#i-would-like-to-refactore
The package was not refactored for a long time. I would like to apply the common invenio structure with the ext.py and proxies.py files. Migrate to setup.cfg and black. Further, as i researched, the classy.py file is not used within inveniosoftware, should we keep it or could we remove it?

#if-we-not-refactore-and-remove-classy
how do we proceed with with the before_first_request decorator in classy.py? In my opinion it is not worth the time to keep that functionality. We could release a new major version which indicates that the classy.py has been removed.

#rename-entrypoint
i would like to rename it to finalize_app_entry_points and then the entrypoint: invenio_base.finalize_app. in my eyes it describes the best what it should do. what do you think?

#structural-problem
there is a problem with the register_menu decorator. This decorator uses the before_first_request decorator and the register_menu decorator is used in 4-5 inveniosoftware packages. In my opinion the register_menu decorator should be removed and the usage of the decorator has to be refactored to use the newly introduced finalize_app_entry_points. My point is, if we keep the register_menu decorator we would have to find a solution where the decorator could still be used, but what the decorator does is done on initializing the app, which seems not possible in my eyes.

SIDENOTE: the register_breadcrumb from flask-breadcrumps is a register_menu wrapper, which means, this has also to be refactored (in the sense of removed)

This change would affect code like in invenio-accounts

SIDENOTE: is this in invenio-administration the example that it is not possible to replace all before_*_request decorators with the entrypoint method?

[slint]

#i-would-like-to-refactore The package was not refactored for a long time. I would like to apply the common invenio structure with the ext.py and proxies.py files. Migrate to setup.cfg and black. Further, as i researched, the classy.py file is not used within inveniosoftware, should we keep it or could we remove it?

+1 for updating the structure and modernizing. What we have to keep in mind is that this is Flask-* package and not invenio-*, so we have to assume that there might be users in the wild. The best would be in any case to release a major bump with breaking changes if that's easier.

#if-we-not-refactore-and-remove-classy how do we proceed with with the before_first_request decorator in classy.py? In my opinion it is not worth the time to keep that functionality. We could release a new major version which indicates that the classy.py has been removed.

We can remove classy.py and support for Flask-Classy, and make a major version bump.

#rename-entrypoint i would like to rename it to finalize_app_entry_points and then the entrypoint: invenio_base.finalize_app. in my eyes it describes the best what it should do. what do you think?

I have a feeling that entrypoints are a bit "esoteric" at this point to how we do things in Invenio... Would it make sense to explore more towards what the discussion in Flask also suggested, i.e. a finalize_init_app method as part of the extensions "interface/convention"? We could then just call that method (if it exists) for each extension in the existing inenio_app.apps and invenio_app.api_apps entrypoints.

#structural-problem there is a problem with the register_menu decorator. This decorator uses the before_first_request decorator and the register_menu decorator is used in 4-5 inveniosoftware packages. In my opinion the register_menu decorator should be removed and the usage of the decorator has to be refactored to use the newly introduced finalize_app_entry_points. My point is, if we keep the register_menu decorator we would have to find a solution where the decorator could still be used, but what the decorator does is done on initializing the app, which seems not possible in my eyes.

This is a bigger problem in the way we integrate Flask-* modules in general in Invenio, since we usually initialize the extensions inside our own InvenioXYZ extensions, and thus deny using the common pattern of accessing methods of an un-initialized extension. See Flask-Talisman as an example, which is technically unusable in its advertised decorator-style usage.

I think we should/could solve this problem as well as part of this refactoring here.

SIDENOTE: the register_breadcrumb from flask-breadcrumps is a register_menu wrapper, which means, this has also to be refactored (in the sense of removed)
This change would affect code like in invenio-accounts

Yes, these two modules are very tightly coupled, and as an extension any module that adds to the user setting pages. I think before releasing any breaking changes we should first pin the dependency in invenio-theme (this is the first place where it's introduced, so thus a "coordinator module").

SIDENOTE: is this in invenio-administration the example that it is not possible to replace all before_*_request decorators with the entrypoint method?

This one looks replaceable actually, but depends on some "local" variables (i.e. view_class), so it might be a bit tricky.

[utnapischtim]

#about-flask-menu
flask-menu depends on flask-classy which is last updated 2014 and superseded by flask-classful (last update 12.2021). flask-classful is not compatible with werkzeug >2.3.0. keeping the classy.py would increase the amount of time to refactore the whole code and make it compatible with werkzeug >=2.3.0 and flask >=2.3.0

the rest tomorrow

[utnapischtim]

@slint
do we still support invenio-admin and invenio-deposit. as i know invenio-admin is deprecated in favor of invenio-administration and invenio-deposit had the last release 2019. i saw that invenio-deposit is a dependency of invenio-github

[utnapischtim]

overview of open PR's all draft.

before(_app)_first_request

• app: add init_functions_loader #172
• app_factory: add init functions entry points invenio-app#88
• fix: before_first_request deprecation invenio-administration#171
• fix: before_first_request deprecation invenio-accounts#440
• fix: before_app_first_request deprecation invenio-app-rdm#2218
• fix: before_first_request deprecation invenio-oauthclient#303
• fix: before_app_first_request deprecation invenio-pages#82
• fix: before_first_request deprecation invenio-communities#958

register_menu decorator PR's

• fix: before_first_request deprecation invenio-userprofiles#161
• fix: before_first_request deprecation invenio-oauth2server#248
• fix: before_first_request deprecation invenio-oauthclient#303

record_once PR's

• fix: before_first_request deprecation invenio-userprofiles#161
• fix: before_first_request deprecation invenio-accounts#440
• fix: before_first_request deprecation invenio-communities#958
• fix: before_first_request deprecation invenio-oauthclient#303

[ntarocco]

@slint do we still support invenio-admin and invenio-deposit. as i know invenio-admin is deprecated in favor of invenio-administration and invenio-deposit had the last release 2019. i saw that invenio-deposit is a dependency of invenio-github.

Correct: no more support to invenio-admin and invenio-deposit.
About invenio-github, I suspect it will be refactored soon (current or next sprint) to enable GitHub integration in the new ZenodoRDM. invenio-deposit will disappear.

If it is easy to fix invenio-admin, I would fix it anyway because we have other running services that still use invenio-admin :)

[utnapischtim]

@slint
how should we do with flask-security-invenio? as the name indicates it is a flask package, but if i add an entrypoint to invenio_base it is not more independent to invenio-* packages.