Logger improvements

[kumare3]

Tracking issue

With this change, flytekit now has 2 loggers, one for framework and one for user code.
Logger is default initialized based on environment. This logger is usually a json logger.

But, when a user runs any pyflyte command, the logger is switched (upgraded) to a rich logger. this can be turned off, but the logging is way more readable. the format is also adjusted

using pyflyte -vvv or pyflyte -v changes verbosity level of the log

 pyflyte -vv run test2.py wf --n 10

The PR also simplifies the logging setup. Now, everywhere in the framework, logger should be used. There is work to be done to make the logging less verbose and more meaningful, that will be done as a follow up

Why are the changes needed?

Better logging when using the CLI

What changes were proposed in this pull request?

Logger setup for flytekit

How was this patch tested?

locally and remotely

[codecov]

Codecov Report

Attention: 38 lines in your changes are missing coverage. Please review.

Comparison is base (6cd0a60) 85.98% compared to head (63f9f1f) 85.97%.
Report is 5 commits behind head on master.

Files	Patch %	Lines
flytekit/remote/remote.py	30.00%	21 Missing ⚠️
flytekit/loggers.py	70.90%	6 Missing and 10 partials ⚠️
flytekit/remote/entities.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2053      +/-   ##
==========================================
- Coverage   85.98%   85.97%   -0.02%     
==========================================
  Files         308      308              
  Lines       22946    22997      +51     
  Branches     3468     3479      +11     
==========================================
+ Hits        19731    19771      +40     
- Misses       2615     2616       +1     
- Partials      600      610      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[honnix]

This is unfortunately a breaking change because we use this handler for further customization. This is the second time this part breaks :(

[eapolinario]

@honnix , sorry for the breaking change. The intent of this change was to simplify the logging infrastructure.

Can you talk about what sort of customizations you are applying to this? We might simply expose that as an escape hatch.

[honnix]

No worries. We have adapted accordingly. So there are two needs:

• we need to customize the json formatter to remap a field (levelname) to satisfy stackdriver logging
• we use the same handler to set all existing loggers (things from other libs) so we can make sure everything is json logged, not just flytekit and user space loggers