Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make connection login and password TEXT #32815

Merged
merged 1 commit into from Sep 26, 2023
Merged

Make connection login and password TEXT #32815

merged 1 commit into from Sep 26, 2023

Conversation

Sighery
Copy link
Contributor

@Sighery Sighery commented Jul 24, 2023

This commit will change the connection login and password fields to be of type TEXT, so it won't have a length limitation anymore.

Closes #32657

@boring-cyborg
Copy link

boring-cyborg bot commented Jul 24, 2023

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Copy link
Member

@hussein-awala hussein-awala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears like MySQL doesn't like your change:

sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')

@Sighery
Copy link
Contributor Author

Sighery commented Jul 25, 2023

It appears like MySQL doesn't like your change:

sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')

Yep. I was talking about it in the related issue. The TL;DR is that it doesn't fail on my migration (128), nor in its downgrade. But rather it fails to downgrade migration 79 after applying that and 128 just fine.

@uranusjr
Copy link
Member

Why do we even have a limit anyway? Since we don’t search against the column we can just use Text.

@Sighery
Copy link
Contributor Author

Sighery commented Jul 25, 2023

Why do we even have a limit anyway? Since we don’t search against the column we can just use Text.

For my usecase that'd also work. I don't think there's an index against either login, password, nor extra. Nor any search functionality for those fields. So I think it would make sense to make them all TEXT.

But I'm new to Airflow, as well as the codebase, so figured I'd open an issue first to ask about it, and then I pushed the code to see if current DB engines would support this change (clearly MySQL doesn't for some reason), and get opinions from Airflow committers/devs.

@eladkal
Copy link
Contributor

eladkal commented Aug 30, 2023

For my usecase that'd also work. I don't think there's an index against either login, password, nor extra. Nor any search functionality for those fields. So I think it would make sense to make them all TEXT.

can you modify the PR for that?

@Sighery
Copy link
Contributor Author

Sighery commented Aug 30, 2023

can you modify the PR for that?

Sure, sounds good. I'm guessing it will require rebasing and getting it up to date now that 2.7.0 is out

@potiuk
Copy link
Member

potiuk commented Aug 30, 2023

Yep. Rebase early-rebase often (as I often repeat)

@Sighery Sighery marked this pull request as draft September 4, 2023 16:00
@Sighery Sighery changed the title Increase connection login length to 5000 Make connection login and password TEXT Sep 14, 2023
@Sighery Sighery marked this pull request as ready for review September 14, 2023 11:41
@Sighery
Copy link
Contributor Author

Sighery commented Sep 14, 2023

@eladkal @potiuk Should be up to date now. Apologies it took me so long to get to it, it just so happened that I was on vacation when the decision for switching to TEXT was made.

I might have missed adding something. If so, just let me know 🙌

@Sighery
Copy link
Contributor Author

Sighery commented Sep 20, 2023

Seems what's failing is TestDatabricksExecutionTrigger.test_sleep_between_retries since the test expects the sleep mock to be called once but it was called 6 times. I tried looking at the test code but I can't get a clear view of how this is connected with these changes. If someone more knowledgeable can take a look, or point me on what to look at to determine if this PR indirectly caused that error, would appreciate it.

This commit will change the connection `login` and `password` fields
to be of type `TEXT`, so it won't have a length limitation anymore.
@hussein-awala hussein-awala merged commit 8e38c5a into apache:main Sep 26, 2023
43 checks passed
@boring-cyborg
Copy link

boring-cyborg bot commented Sep 26, 2023

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@hussein-awala
Copy link
Member

Congrats on your first commit 🎉

@Sighery Sighery deleted the increase-connection-login-length-5000 branch September 27, 2023 14:36
@ephraimbuddy ephraimbuddy added this to the Airflow 2.7.2 milestone Oct 3, 2023
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Oct 3, 2023
@ephraimbuddy ephraimbuddy added type:improvement Changelog: Improvements and removed type:bug-fix Changelog: Bug Fixes labels Oct 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Increase connections HTTP login length to 5000 characters
6 participants