[FEATURE] Requesting brickflow support for python versions 3.11 and 3.12

[raajaaDheeRaajaa]

Is your feature request related to a problem? Please describe.
Currently, the compatible versions of python for brickflow are pinned to 3.8, 3.9 & 3.10

brickflow/pyproject.toml

Line 37 in 09a2af9

python = ">=3.8,<3.11" # pyspark <3.5 does not play happy with python 3.11

Requesting to add support for python versions 3.11 & 3.12, for users using pyspark version >=3.5

Cloud Information

• AWS
• Azure
• GCP
• Other

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
Updating the acceptable python version from >=3.8,<3.11 to >=3.8,<=3.12 here:

brickflow/pyproject.toml

Line 37 in 09a2af9

python = ">=3.8,<3.11" # pyspark <3.5 does not play happy with python 3.11

Additional context
This change will allow users running newer version of python with new features and bug fixes, able to use brickflow.

[t1ffanyt]

Hello, I can take up this issue and test brickflow with the updated python versions.

[stikkireddy]

Hey team, intentially we are supporting 3.8 - 3.10 as those are the system versions supported by databricks runtimes.
14.3 LTS beta is provided with Python: 3.10.12
https://docs.databricks.com/en/release-notes/runtime/14.3lts.html#system-environment

The risk of supporting higher versions is it may encourage users to use newer 3.11, 3.12 and 3.13 features which may not work on 3.10. The user will not run into this without, actually running their workflow.

I am not opposed to this change, as the library should work in higher versions (since new versions of python contain all the previous features) but I do not know of a good way to indicate to the user to not use 3.11+ features. Let me know if you have any ideas around that.