Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid pendulum.from_timestamp usage #37160

Merged
merged 1 commit into from Feb 3, 2024

Conversation

Taragolis
Copy link
Contributor

This function use deprecated in Python 3.12 datetime.datetime.utcfromtimestamp sdispater/pendulum#803 as result it might cause recursion error into the Airflow logger in Python 3.12

2024-02-01T00:16:09.5682821Z /usr/local/lib/python3.12/site-packages/pendulum/__init__.py:295 DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
2024-02-01T00:16:09.5683171Z Process DagFileProcessor109-Process:
2024-02-01T00:16:09.5683635Z Traceback (most recent call last):
2024-02-01T00:16:09.5684622Z   File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
2024-02-01T00:16:09.5684847Z     self.run()
2024-02-01T00:16:09.5685455Z   File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
2024-02-01T00:16:09.5685746Z     self._target(*self._args, **self._kwargs)
2024-02-01T00:16:09.5686591Z   File "/opt/airflow/airflow/dag_processing/processor.py", line 182, in _run_file_processor
2024-02-01T00:16:09.5686850Z     _handle_dag_file_processing()
2024-02-01T00:16:09.5687635Z   File "/opt/airflow/airflow/dag_processing/processor.py", line 161, in _handle_dag_file_processing
2024-02-01T00:16:09.5688168Z     log.info("Started process (PID=%s) to work on %s", os.getpid(), file_path)
2024-02-01T00:16:09.5688718Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1539, in info
2024-02-01T00:16:09.5688970Z     self._log(INFO, msg, args, **kwargs)
2024-02-01T00:16:09.5689526Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
2024-02-01T00:16:09.5689738Z     self.handle(record)
2024-02-01T00:16:09.5690307Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1700, in handle
2024-02-01T00:16:09.5690536Z     self.callHandlers(record)
2024-02-01T00:16:09.5691167Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
2024-02-01T00:16:09.5691378Z     hdlr.handle(record)
2024-02-01T00:16:09.5691947Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1028, in handle
2024-02-01T00:16:09.5692169Z     self.emit(record)
2024-02-01T00:16:09.5692799Z   File "/opt/airflow/airflow/utils/log/file_processor_handler.py", line 74, in emit
2024-02-01T00:16:09.5693038Z     self.handler.emit(record)
2024-02-01T00:16:09.5693571Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1280, in emit
2024-02-01T00:16:09.5693830Z     StreamHandler.emit(self, record)
2024-02-01T00:16:09.5694362Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1160, in emit
2024-02-01T00:16:09.5694614Z     msg = self.format(record)
2024-02-01T00:16:09.5694811Z           ^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5695368Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 999, in format
2024-02-01T00:16:09.5695598Z     return fmt.format(record)
2024-02-01T00:16:09.5695803Z            ^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5696632Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 705, in format
2024-02-01T00:16:09.5697033Z     record.asctime = self.formatTime(record, self.datefmt)
2024-02-01T00:16:09.5697291Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5697925Z   File "/opt/airflow/airflow/utils/log/timezone_aware.py", line 42, in formatTime
2024-02-01T00:16:09.5698456Z     dt = pendulum.from_timestamp(record.created, tz=pendulum.local_timezone())
2024-02-01T00:16:09.5698748Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5700066Z   File "/usr/local/lib/python3.12/site-packages/pendulum/__init__.py", line 295, in from_timestamp
2024-02-01T00:16:09.5700446Z     dt = _datetime.datetime.utcfromtimestamp(timestamp)
2024-02-01T00:16:09.5700702Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5701257Z   File "/usr/local/lib/python3.12/warnings.py", line 109, in _showwarnmsg
2024-02-01T00:16:09.5701655Z     sw(msg.message, msg.category, msg.filename, msg.lineno,
2024-02-01T00:16:09.5702224Z   File "/opt/airflow/airflow/settings.py", line 126, in custom_show_warning
2024-02-01T00:16:09.5702530Z     write_console.print(msg, soft_wrap=True)
2024-02-01T00:16:09.5703346Z   File "/usr/local/lib/python3.12/site-packages/rich/console.py", line 1673, in print
2024-02-01T00:16:09.5703554Z     with self:
2024-02-01T00:16:09.5704351Z   File "/usr/local/lib/python3.12/site-packages/rich/console.py", line 865, in __exit__
2024-02-01T00:16:09.5704578Z     self._exit_buffer()
2024-02-01T00:16:09.5705422Z   File "/usr/local/lib/python3.12/site-packages/rich/console.py", line 823, in _exit_buffer
2024-02-01T00:16:09.5705887Z     self._check_buffer()
2024-02-01T00:16:09.5706785Z   File "/usr/local/lib/python3.12/site-packages/rich/console.py", line 2060, in _check_buffer
2024-02-01T00:16:09.5707003Z     self.file.write(text)
2024-02-01T00:16:09.5707585Z   File "/opt/airflow/airflow/utils/log/logging_mixin.py", line 200, in write
2024-02-01T00:16:09.5707934Z     self.flush()
2024-02-01T00:16:09.5708527Z   File "/opt/airflow/airflow/utils/log/logging_mixin.py", line 207, in flush
2024-02-01T00:16:09.5708752Z     self._propagate_log(buf)
2024-02-01T00:16:09.5709415Z   File "/opt/airflow/airflow/utils/log/logging_mixin.py", line 188, in _propagate_log
2024-02-01T00:16:09.5709816Z     self.logger.log(self.level, remove_escape_codes(message))
2024-02-01T00:16:09.5710466Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1609, in log
2024-02-01T00:16:09.5710730Z     self._log(level, msg, args, **kwargs)
2024-02-01T00:16:09.5711279Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
2024-02-01T00:16:09.5711498Z     self.handle(record)
2024-02-01T00:16:09.5712065Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1700, in handle
2024-02-01T00:16:09.5712306Z     self.callHandlers(record)
2024-02-01T00:16:09.5712912Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
2024-02-01T00:16:09.5713142Z     hdlr.handle(record)
2024-02-01T00:16:09.5713697Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1028, in handle
2024-02-01T00:16:09.5713912Z     self.emit(record)
2024-02-01T00:16:09.5714543Z   File "/opt/airflow/airflow/utils/log/file_processor_handler.py", line 74, in emit
2024-02-01T00:16:09.5714777Z     self.handler.emit(record)
2024-02-01T00:16:09.5715311Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1280, in emit
2024-02-01T00:16:09.5715574Z     StreamHandler.emit(self, record)
2024-02-01T00:16:09.5716103Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 1160, in emit
2024-02-01T00:16:09.5716346Z     msg = self.format(record)
2024-02-01T00:16:09.5716539Z           ^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5717093Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 999, in format
2024-02-01T00:16:09.5717319Z     return fmt.format(record)
2024-02-01T00:16:09.5717525Z            ^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5718074Z   File "/usr/local/lib/python3.12/logging/__init__.py", line 705, in format
2024-02-01T00:16:09.5718464Z     record.asctime = self.formatTime(record, self.datefmt)
2024-02-01T00:16:09.5718719Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5719349Z   File "/opt/airflow/airflow/utils/log/timezone_aware.py", line 42, in formatTime
2024-02-01T00:16:09.5719890Z     dt = pendulum.from_timestamp(record.created, tz=pendulum.local_timezone())
2024-02-01T00:16:09.5720173Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.5721122Z   File "/usr/local/lib/python3.12/site-packages/pendulum/__init__.py", line 295, in from_timestamp

... truncated 2000+ lines of traceback

2024-02-01T00:16:09.6686905Z     self.logger.log(self.level, remove_escape_codes(message))
2024-02-01T00:16:09.6687301Z                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.6688023Z   File "/opt/airflow/airflow/utils/log/logging_mixin.py", line 61, in remove_escape_codes
2024-02-01T00:16:09.6688266Z     return ANSI_ESCAPE.sub("", text)
2024-02-01T00:16:09.6688481Z            ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.6689163Z   File "/usr/local/lib/python3.12/site-packages/re2.py", line 291, in sub
2024-02-01T00:16:09.6689496Z     joined_pieces, _ = self.subn(repl, text, count)
2024-02-01T00:16:09.6689733Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-02-01T00:16:09.6690066Z RecursionError: maximum recursion depth exceeded

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis added this to the Airflow 2.8.2 milestone Feb 3, 2024
@Taragolis Taragolis merged commit 8374105 into apache:main Feb 3, 2024
82 checks passed
@Taragolis Taragolis deleted the ban-pendulum-from-timestamp branch February 3, 2024 21:51
potiuk pushed a commit that referenced this pull request Feb 7, 2024
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Feb 19, 2024
ephraimbuddy pushed a commit that referenced this pull request Feb 22, 2024
abhishekbhakat pushed a commit to abhishekbhakat/my_airflow that referenced this pull request Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants