[Feat][Doc] using poetry for managing all the python dependencies

[SemyonSinchenko]

Is your feature request related to a problem? Please describe.
Currently python dependencies are managed in two ways:

1. PySpark poetry project
2. requirements.txt + pip for sphinx docs generation

Describe the solution you'd like
poetry as a single system for all the python dependencies

Describe alternatives you've considered
.

Additional context
As discussed in mailing list and on the community meeting.