[AIRFLOW-6838][WIP] Introduce real subcommands for Breeze

[mik-laj]

Breeze currently selects action using many different flags. I know from my own experience that this is very confusing. I think that subcommands will be much easier to use.

I suspect that Breeze autocomplete is not working properly, but I have no experience with this component. Could someone help me with this?

In the next step, I think that we should replace global variables with functions, because a large number of global variables makes it very difficult to trace the aapplication behavior.

---

Issue link: AIRFLOW-6838

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[potiuk]

While I like the spirit of that change,. Breeze grew a lot in terms of functionality and maybe we should think of refactoring the structure of commands soon.

But I think it's not the right way of doing it.

First of all you are just about to have a lot of conflicts as we are splitting breeze into functions. Secondly it does not take into account getopt's behaviour so it is for sure wrong (We define all the available options in breeze-complete and getopt by default assumes -- or - params. So not only autocomplete will not work but also parsing options won't work either.

I think we need to do much bolder change if we want to introduce sub-commands - we should rewrite it in another language (python or golang). But complicating parsing of parameters is not a good idea.

Unless you use ready to use library - for example here http://people.tuebingen.mpg.de/maan/gsu/ then we could write it in bash (but it's not a good idea as it is GPL2 and it does not support ZSH I believe). I think we should talk about it and think if that's the right time to think of such change or maybe better to leave it for after we deliver the current important 2.0 features - prod image, API and others.

[potiuk]

IT's not terminal. It's the whole environment using integrations, runtime, kubernetes, databases etc. I still think environment is better name,

[mik-laj]

also parsing options won't work either.

Can you say something more about it? I checked, I did found any issues

[potiuk]

Sure: getopt verifies switches provided and will complain if the switch is wrong. Try --nnn . with the command specified you get no verification you can do ./breeeze nnnn and it will do something instead of complaining. If you specify --stop-environment as switch, it will pass over getopt but --stop-environments will fail in get_opt. And the list of options (available in breeze-complete) is not updated. The same list is used for autocomplete

[potiuk]

Even if you provide --stop, get opt will suggest that it is ambigius and will tell you that --stop-environment is one of possible options.

[mik-laj]

I updated autocompllete. This should also solve the problems with getopt.

you can do ./breeeze nnnn and it will do something instead of complaining.

Thanks. I added missing check.

I still have to do rebase, but I will do it tomorrow.

[potiuk]

This is much better now. I really like where it goes.

Two general comments:

• I think we should get rid of the short one-letter options for sub-commands. It will make it a bit smaller and I think it makes much more sense in the context of this change. I've never seen one letter commands for subcommands in any other tool

• we should behave similarly as other tools using subcommands:
  ./breeze --help should return only list of subcommands with short description, common options for all commands
  ./breeze <subcommand> --help should return detailed explanation for that command only

I am happy to work together with you on that and contribute a commit or two to add it after you rebase your changes.

[mik-laj]

@potiuk I rebased. Can you look at it?

[mik-laj]

I think we should get rid of the short one-letter options for sub-commands.

Removed

./breeze --help should return detailed explanation for that command only

I agree, but I think this is an additional function that we can do independently. This will require much more code changes. In my opinion, we can divide it into two steps. First enter the commands and check users' opinions about them, and then refactor the code to further improve the code.

This is a big change, because it will also require that we do not have one parser, but several smaller parsers.

[potiuk]

I think we should get rid of the short one-letter options for sub-commands.

Removed

they're still there.

[potiuk]

This is a big change, because it will also require that we do not have one parser, but several smaller parsers.

Not necessary. that's just splitting help - nothing else. One parser should be enough. Again - happy to add it.

[mik-laj]

they're still there.

pre-commit got stuck when asking to rebuild the image. I pushed the changes now.

Not necessary. that's just splitting help - nothing else. One parser should be enough. Again - happy to add it.

I invite you to contribute.

[potiuk]

Will do !

[mik-laj]

Done: #7515