Future of Flask-Script with Flask 1.0

[mitsuhiko]

I'm not sure how I want to do the transition myself yet but I'm more than happy with how Click (http://click.pocoo.org/) (and the integration of it into Flask 1.0) is going that I would like this to be the official way forward to build command line utilities in the Flask ecosystem.

My plan is that by next month there is a new Flask release that supports Click out of the box and recommends using it.

Given how many extensions currently use Flask-Script and how popular it became I would like to figure out how to migrate everything over to Click without causing a lot of frustration everywhere.

First things first: the click integration in Flask is barebones out of the box. It does have a run and shell command but it does not do anything fancy like ipython integration or printing url maps. However it does support loading in extra commands from extensions so a Flask-Script successor could provide all that missing functionality.

To answer the obvious questions why not Flask-Script:

• Flask-Script is old and predates two major changes in Flask. The main problem with that is that Flask-Script sets up a request context even though there is really on reason to do this.
• Flask-Script loads the app very early which causes problems for the reloader when developing. Primarily it means that a syntax error will cause the reloader to crash. It also means that with the reloader active, top level code runs twice. Once in the oute rand once in the inner python process. Flask's click integration solves this.

Unfortunately these two design differences mean there is no clear migration path.

I would like to figure out how to progress best and how to make this as painless as possible.

[rochacbruno]

I use Flask-Script in several projects, so I hope to have a "compatibility-mode" maybe click or Flask-Script could have the ability to transform commands.

The best scenario is click having a click.register_flask_script_command(Command_Class) or click.import_from_flask_script(Manager_Instance), but can work also on reverse manager.command.as_click_command()

[hvdklauw]

Not clear on all the internals of both click and flask-script but it would be nice to have something in flask-script that can convert from one to the other.

I do know however that with flask-script I wrote commands sometimes that took a app factory and would allow for command line parameters/options to be passed to the factory.
Which was a really nice feature and haven't figured out yet how to do with click

[mitsuhiko]

I do know however that with flask-script I wrote commands sometimes that took a app factory and would allow for command line parameters/options to be passed to the factory.

That is supported when you write your own scripts with click and hook them up with your Flask application instead of using the default flask script. Example from the docs:

import click
from flask.cli import FlaskGroup, script_info_option

def create_wiki_app(info):
    from yourwiki import create_app
    config = info.data.get('config') or 'wikiconfig.py'
    return create_app(config=config)

@click.group(cls=FlaskGroup, create_app=create_wiki_app)
@script_info_option('--config', script_info_key='config')
def cli(**params):
    """This is a management script for the wiki application."""

if __name__ == '__main__':
    cli()

A compatibility mode is what I would like to see, but not in Click (since it not specific to Flask) and not in Flask, because then it opens the doors to be compatible to Flask-Actions and i don't know what. It also means that Flask could carry the compatibility burden for every single developer even if they never used Flask-Script.

[miguelgrinberg]

@mitsuhiko I looked at your Click integration layer. It seems the code that addresses the two Flask-Script design limitations you mention above is not Click specific at all.

In fact, in just a few minutes of hacking I was able to get Flask-Script to work with them as well. All I used from cli.py is class DispatchingApp and function locate_app. I just had to add a couple of methods to DispatchingApp, but I did not need to make any changes to Flask-Script, which still calls app.run() to start.

As I mentioned yesterday on reddit, I feel adding yet another dependency does not agree with the lightweight spirit of Flask, so here is my proposal:

• Move the Click integration to a Flask-Click extension. I would make the extension work on older Flask releases, with the known caveat that the reloader is broken on anything older than Flask 1.0.
• Keep the delayed app loading functionality in Flask, but cleaned up so that it can be used by Flask-Click, Flask-Script or standalone (if you just want the reloader w/o cli support, see next two items).
• Remove use_reloader from app.run(), or document it is broken when used this way.
• Make __main__.py be a simple reloader wrapper for an app module w/o args, instead of a full blown command line.

If exploring this option is something that interests you I'm happy to help. Thanks.

[mitsuhiko]

@mitsuhiko I looked at your Click integration layer. It seems the code that addresses the two Flask-Script design limitations you mention above is not Click specific at all.

You can solve this issue in many different ways. All you need to do is to avoid the app being loaded. Flask's CLI command did not use click initially but the initial version of what I had I could not get to work with Flask-Script at all. Now that I went through a bunch of iterations I would not rule out that you can't use it with Flask-Script.

As a general hint: Deprecating Flask-Script is not something that needs to happen, but at present I'm pretty sure Flask will ship click integration of some sort.

Make main.py be a simple reloader wrapper for an app module w/o args, instead of a full blown command line.

But what would it gain from that? Yes, there would be one dependency less but it's non critical dependency really. flask.cli was initially not using Click but then there was a one-trick pony installed with Flask and it was written in a way that Flask-Script could not take advantage of it much and the complexity of that implementation is probably higher than the click integration. At least the initial implementation was not exactly easier: https://github.com/mitsuhiko/flask/blob/1b06c8a41119f8c93078e85f6474cb85d7e30b43/flask/run.py

[miguelgrinberg]

but the initial version of what I had I could not get to work with Flask-Script at all

In terms of the reloader fix that first version looks pretty similar to what you have in the latest version. To make things work with Flask-Script I had to initialize the Manager instance with a factory function, obviously you cannot use the app directly. The factory function that I used returns a DispatchingApp instance that references the module that contains the application. I added a run() method to DispatchingApp that loads the app if it isn't there yet and then passes the call on to it. This makes Flask-Script's runserver option work.

I also added a DispatchingApp.__getattribute__ method that acts as a proxy, so that any time someone tries to access a property of the app the app is loaded first. I feel this is important to ensure that legacy code continues to work.

Regardless of what you do with Click, I suggest DispatchingApp is documented and made available with Flask 1.0, so that it can be used by other extensions, or to have reloader support access through scripting instead of a cli.

As a general hint: Deprecating Flask-Script is not something that needs to happen

Sure, it doesn't need to happen. But you also made the following statement:

I would like to figure out how to migrate everything over to Click without causing a lot of frustration everywhere.

So it seems the message you will give is that Flask-Script should be replaced with Click. So the moment Flask 1.0 is out my extension will be perceived as outdated.

As an extension developer with dependencies on Flask-Script I think I have three options:

1. I can say f*** it, I'll continue working with Flask-Script. Then people working on Flask 1.0 will be forced to also use Flask-Script against your recommendation. Eventually someone will also say f*** it and will take my extension and make a Click version of it. So now we have duplicated extensions, just because of silly dependency problems.
2. I can say f*** the older versions, I'll migrate my extensions to Click and work with 1.0 and up. So I'll be forcing all my users who for one reason or another do not want or can't upgrade to 1.0 to stay on an older version of my extension. Dependency checking will be messy, it can be documented, but I'm sure you agree that people will make mistakes. And extension developers will get the bugs.
3. I could think hard and figure out a way to support Flask-Script for Flask < 1.0 and Click for Flask >= 1.0, all in the same extension. Assuming this can be made to work then I think it would cause the least trouble to users, but it is a pain for extension developers. I certainly do not want do this.

[fgimian]

Hey guys, just thought I'd share some of my thoughts too if I may.

Firstly, I think that click is awesome! I have already used it in a project of my own and will definitely continue to use it over argparse in the future. Congratulations on a great job done there Armin! 😄

The integration of a script interface based on click within Flask is something that I'm personally very excited about, let alone the release of Flask 1.0 which is even more exciting. The management script interface is something that I personally always use on every project anyway so it would be great if it was part of the core. If anything, I'd love to see the Flask-Cache extension also included in the core too 😄

I understand why some may have reservations about including more dependencies in Flask, but imho as long as the ORM stays separate, then Flask is still a very loosely coupled framework in almost all respects.

All the best
Fotis

[rsyring]

As an interested observer, my $.02 is as follows:

• I like the idea of having one method of doign the CLI and like the fact that Click is going to be built into Flask. While I apprecaite the light-weight nature of Flask, I think this is a win.
• You could deprecate Flask-Script but keep it around because of legacy users. Then,
• Fork Flask-Script as Flask-CLIPlus or Flask-ClickPlus, etc. The main thing being you change the name of the project. Now that Flask is going to come with Click for scripting support, changing the name of the project to indicate that it is providing enhanced features seems to make sense to me and I think would make sense to most devs. Put warnings on the Flask-Script docs pointing to Flask-CLIPlus as the successor, due to Click integration in Flask, and it will make sense to most devs.

IMO, this is essentially your option #2 above, with a project name change, and without all the hassle of worrying about dependencies, legacy users, etc.

[reubano]

Instead of making a new Flask-Click extension, why not just update this project?

[harobed]

@reubano I think because too many incompatibility.

[reubano]

@harobed I don't think that's the case. Based on @miguelgrinberg comments above, he says he's been able to port over the fixes.

[ThiefMaster]

@miguelgrinberg: Your comment is a bit old, but I think the proper solution for that problem is to simply drop support for Flask<1.0 in a new version of your extension. That way nothing will break, and nobody can expect newer versions of extensions to keep supporting older versions of Flask.

[miguelgrinberg]

@ThiefMaster Yes, that's my #2 option above. What I don't like about that is that I need to create an arbitrary major version change in my project because a dependency is forcing me to. Doable, yes. Do I like it? No.

[lu-zero]

@miguelgrinberg how hard would be to have Flask-Script wrap Click so it is completely transparent to the user?

[ThiefMaster]

That would sacrify all the nice things click provides. Also, one would need to write a tool that converts for argparse-style command/argument definitions to use the click API. Probably not worth it..

[miguelgrinberg]

@lu-zero it's a non-trivial effort, but in any case, what's the point of doing it? If we are going to make click work exactly like Flask-Script in a transparent way, then why not continue using Flask-Script instead?

What a lot of people seem to forget is that we are talking about command line arguments here. It's simple stuff that has been figured out a long time ago. Yes, click has an original way of defining command line arguments that is nice and even fun to use, but at the end of the day, for me command line arguments in a Flask app are a super tiny fraction of my worries, honestly it makes no difference to me if I use click or Flask-Script because both make the task easy, and Flask-Script has the advantage that is compatible with argparse, which click is not.

In my view Flask has no say in what command line parser I use, so if it was up to me, I would not make it a dependency. This goes against the no-batteries spirit of Flask, in my opinion. There is a claim that click resolves problems that are hard or impossible to address in Flask-Script (see top of this thread), but I demonstrated long ago that this isn't true, with a few small changes in Werkzeug the reload works with Flask-Script as well as it works with click.

[rlam3]

@miguelgrinberg what versions of flask-script works best with which version of flask?

I'm still wondering what your thoughts are now as well as to whether or not old flask users should be migrating over to click as well... what would a flask-script migrate to click look like?

maybe a tutorial would help for those who would like to migrate. thanks!

Also, I see a new way to migrate db using click as well....your thoughts?
https://github.com/eatfirst/Flask-Click-Migrate

[ThiefMaster]

@rlam3: Flask-Migrate already registers a click subcommand (flask db) via its setup.py. There shouldn't be any need for a separate package.

miguelgrinberg/Flask-Migrate@8c208b5

[miguelgrinberg]

what versions of flask-script works best with which version of flask?

I never saw incompatibilities, as far as I know Flask-Script continues to work well with Flask, even the releases that use click.

whether or not old flask users should be migrating over to click as well

Up to you entirely. Flask-Script still works, so if you have no issues with it, you may very well decide to stick to it. But if you have some free time and would like to migrate, it isn't that difficult, so go for it.

The click interface has an old reloader bug fixed. If you tend to write too many typos in your Flask code and that makes your reloader crash, then switching to click addresses that issue. That is the only significant difference between the two, in my opinion.

Also, I see a new way to migrate db using click as well

No need to use that. I have added support for click on my extensions. Flask-Migrate works with both Flask-Script and click at this time.

[rlam3]

@miguelgrinberg

Is there a way to find out/debug as to why manage.py when using flask-script server isn't detecting changes in my code?

I have tried the following:
reinstalling pip packages on a seperate virtualenv
installing the latestest build from git for flask-script https://github.com/smurfix/flask-script.git
export FLASK_DEBUG=1

Environment

Mac Sierra: 10.12.3
python: Python 2.7.10 // in virtualenv
editor: Atom 1.13.0 x64

#requirements.txt
alabaster==0.7.9
alembic==0.8.5
amqp==1.4.7
anyjson==0.3.3
apache-libcloud==0.17.0
argh==0.26.1
arpy==1.1.1
Babel==2.3.4
backports.ssl-match-hostname==3.4.0.2
billiard==3.3.0.21
blinker==1.3
boto==2.38.0
celery==3.1.19
certifi==14.5.14
cffi==1.9.1
click==6.6
contextlib2==0.5.4
coverage==3.7.1
decorator==3.4.0
defusedxml==0.4.1
depot==0.0.12
docopt==0.6.2
docutils==0.13.1
ecdsa==0.13
elasticsearch==2.3.0
elasticsearch-dsl==2.0.0
enum34==1.1.6
esengine==0.0.18
Fabric==1.13.1
fabtools==0.19.0
filedepot==0.2.1
Flask==0.12
Flask-Babel==0.9
Flask-CDN==1.4.0
Flask-Celery-Helper==1.1.0
Flask-DebugToolbar==0.10.0
Flask-JSGlue==0.3
Flask-JWT-Extended==1.1.0
Flask-Login==0.3.2
Flask-Mail==0.9.1
flask-marshmallow==0.7.0
Flask-Migrate==1.8.0
Flask-OpenID==1.2.4
Flask-Principal==0.4.0
Flask-Script==2.0.6
Flask-Security==1.7.4
Flask-SQLAlchemy==2.1
Flask-SSLify==0.1.5
Flask-Test==0.1.5
Flask-Testing==0.4.2
Flask-WTF==0.13.1
flipflop==1.0
flower==0.8.2
futures==2.2.0
gnureadline==6.3.3
guess-language==0.2
idna==2.2
imagesize==0.7.1
infinity==1.3
intervals==0.7.1
ipaddress==1.0.18
ipython==3.2.0
itsdangerous==0.24
Jinja2==2.9.4
kombu==3.0.29
livereload==2.3.2
lockfile==0.10.2
Mako==1.0.4
MarkupSafe==0.23
marshmallow==2.10.5
marshmallow-sqlalchemy==0.8.1
migrate==0.2.2
nose==1.3.7
paramiko==2.1.1
passlib==1.6.2
pathtools==0.1.2
pbr==0.10.8
phonenumbers==7.0.7
Pillow==3.4.1
pip-tools==0.3.6
psycopg2==2.6
pyasn1==0.1.9
pycountry==1.12
pycparser==2.17
pycrypto==2.6.1
Pygments==2.1.3
PyJWT==1.4.2
pyOpenSSL==0.15.1
python-dateutil==2.5.2
python-editor==1.0
python-gnupg==0.3.7
python-openid==2.2.5
pytz==2016.10
PyYAML==3.11
raven==5.32.0
requests==2.12.5
simplekv==0.10.0
six==1.10.0
snowballstemmer==1.2.1
speaklater==1.3
Sphinx==1.5.1
sphinx-rtd-theme==0.1.9
SQLAlchemy==1.1.1
sqlalchemy-migrate==0.9.5
SQLAlchemy-Utils==0.31.6
sqlparse==0.1.14
stripe==1.22.3
Tempita==0.5.2
tornado==4.1
Unidecode==0.4.18
urllib3==1.14
uWSGI==2.0.14
validators==0.10
watchdog==0.8.3
Werkzeug==0.11.15
WTForms==2.1
WTForms-Alchemy==0.15.0
WTForms-Components==0.10.0
WTForms-SQLAlchemy==0.1

#manage.py

from flask_script import Manager, Shell, Server
from myapp.application import create_app
manager = Manager(create_app())
manager.add_command("runserver",
    Server(
        threaded=True,
        use_reloader=manager.app.debug,
        use_debugger=manager.app.debug,
        host='0.0.0.0',
        port=5000,
    )
)

if __name__ == '__main__':
    manager.run()

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with fsevents reloader
 * Debugger is active!
 * Debugger pin disabled.  DEBUGGER UNSECURED!

The click interface has an old reloader bug fixed. If you tend to write too many typos in your Flask code and that makes your reloader crash, then switching to click addresses that issue. That is the only significant difference between the two, in my opinion.

Could this be why i may be having this issue?

would the version of click or some other package be preventing me from detecting the code changes?
Where would I be able to single out why this is? Thanks!

[davidism]

A discussion of the direction of Click vs Flask-Script isn't really the place to be debugging your code, @rlam3. I can't reproduce your issue. Code reloading works, in both systems. If it doesn't work for you, please use Stack Overflow for questions about your own code. Or open a bug report about this specific issue if you truly feel it's a bug with Flask-Script. Either way, be sure to include a Minimal, Complete, and Verifiable Example.