Skip to content

feat(python): update to python-build-standalone 3.13.2#26125

Merged
jdstrand merged 6 commits intomainfrom
jdstrand/use-python-standalone-3.13
Mar 14, 2025
Merged

feat(python): update to python-build-standalone 3.13.2#26125
jdstrand merged 6 commits intomainfrom
jdstrand/use-python-standalone-3.13

Conversation

@jdstrand
Copy link
Contributor

@jdstrand jdstrand commented Mar 11, 2025

Closes #26044

I need to do (essentially) a full round of testing for #26124, so figured I could use that testing to test this upgrade. As such, my plan is:

This change will cause all current venvs to stop working and require them to be recreated because we don't have a process for migrating existing venvs (see #26045 and #26062). That said, doing this now allows us to get this change in before beta and it will help avoid issues like #26062 for security updates (though it doesn't solve everything; still need what I outlined in the aforementioned issues).

If there are regressions with 3.13 in my testing, I'll revert.

@jdstrand jdstrand force-pushed the jdstrand/use-python-standalone-3.13 branch from 0cbe235 to 274d1f6 Compare March 11, 2025 19:14
@jdstrand
Copy link
Contributor Author

jdstrand commented Mar 11, 2025

If there are regressions with 3.13 in my testing, I'll revert.

Hmm, circleci/test is failing: https://app.circleci.com/pipelines/github/influxdata/influxdb/44648/workflows/d7742273-d45e-424a-b001-b5929d7da37e/jobs/422585

Excerpt:

        FAIL [   0.057s] influxdb3_processing_engine tests::test_trigger_lifecycle
──── STDOUT:             influxdb3_processing_engine tests::test_trigger_lifecycle

running 1 test
──── STDERR:             influxdb3_processing_engine tests::test_trigger_lifecycle
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff5f76006c0 (most recent call first):
  <no Python frame>

error: test run failed

We are fetching the correct python build standalone according to Extract python for this target and PYO3_CONFIG_FILE is correct:

implementation=CPython
version=3.13
shared=true
abi3=false
lib_name=python3.13
lib_dir=/tmp/workspace/python-artifacts/x86_64-unknown-linux-gnu/python/lib
executable=/tmp/workspace/python-artifacts/x86_64-unknown-linux-gnu/python/bin/python3.13
pointer_width=64
build_flags=
suppress_build_script_link_lines=false

I've seen this encodings issue before and it usually means that we aren't setting up the runtime env correctly, which is what https://github.com/influxdata/influxdb/blob/main/influxdb3_processing_engine/src/virtualenv.rs#L108 does. Are we doing (the equivalent of) this in tests? If so, then perhaps something in 3.13 has changed; if not, I think this test target might be using the system libpython which in the build env is libpython3.11 which is the same MAJOR.MINOR that we are using for python-build-standalone prior to this PR and we've been lucky it's working.

@jacksonrnewhouse - does anything here ring a bell (I'm still focused on the glibc portability work so if not, I'll just close this PR).

@jdstrand jdstrand added the v3 label Mar 11, 2025
@jdstrand jdstrand force-pushed the jdstrand/use-python-standalone-3.13 branch from a0a5f47 to 7506014 Compare March 12, 2025 18:54
@jdstrand
Copy link
Contributor Author

@jacksonrnewhouse - are you still looking at this? This is a breaking change (due to not having venv migrate functionality) and if it isn't going to land before beta we should probably close it (cc @dgnorton)

@jacksonrnewhouse jacksonrnewhouse force-pushed the jdstrand/use-python-standalone-3.13 branch from d6b6225 to 00be390 Compare March 14, 2025 17:58
@jacksonrnewhouse jacksonrnewhouse force-pushed the jdstrand/use-python-standalone-3.13 branch from e3d3c35 to 4b04222 Compare March 14, 2025 19:27
Copy link

@jacksonrnewhouse jacksonrnewhouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think this is working now, lets get it into Beta.

@jdstrand
Copy link
Contributor Author

Okay, I think this is working now, lets get it into Beta.

Thanks @jacksonrnewhouse!

Not for this PR but the timeouts in the tests make me wonder if this will be flaky. Perhaps it could be updated to poll for readiness in some manner?

That said, merging as is.

@jdstrand jdstrand merged commit ba7e6c6 into main Mar 14, 2025
12 checks passed
@jdstrand
Copy link
Contributor Author

jdstrand commented Mar 14, 2025

Okay, I think this is working now, lets get it into Beta.

Thanks @jacksonrnewhouse!

Not for this PR but the timeouts in the tests make me wonder if this will be flaky. Perhaps it could be updated to poll for readiness in some manner?

That said, merging as is.

I forgot to mention that I tested this locally too (but with @jacksonrnewhouse's changes for python by default and fixes for the tests, we have tests everywhere during the builds, woohoo!!):

$ /home/jamie/tmp/influxdb3-pe/install/influxdb3 test schedule_plugin -d mydb01 test-schedule.py 
{
  "trigger_time": "2025-03-14T20:19:40Z",
  "log_lines": [
    "INFO: _tst_log(): success",
    "INFO: _tst_sys(): sys.prefix = /home/jamie/tmp/influxdb3-pe/data/plugins/.venv",
    "INFO: _tst_sys(): sys.exec_prefix = /home/jamie/tmp/influxdb3-pe/data/plugins/.venv",
    "INFO: _tst_sys(): sys.base_prefix = /home/jamie/tmp/influxdb3-pe/install/python",
    "INFO: _tst_sys(): sys.base_exec_prefix = /home/jamie/tmp/influxdb3-pe/install/python",
    "INFO: _tst_sys(): sys.path = ['/home/jamie/tmp/influxdb3-pe/install/python/lib/python313.zip', '/home/jamie/tmp/influxdb3-pe/install/python/lib/python3.13', '/home/jamie/tmp/influxdb3-pe/install/python/lib/python3.13/lib-dynload', '/home/jamie/tmp/influxdb3-pe/data/plugins/.venv/lib/python3.13/site-packages']",
    "INFO: _tst_wheel(): 'requests' location = /home/jamie/tmp/influxdb3-pe/data/plugins/.venv/lib/python3.13/site-packages/requests/__init__.py",
    "INFO: _tst_wheel(): requests = 2.32.3",
    "INFO: _tst_libssl(): 'ssl' location = /home/jamie/tmp/influxdb3-pe/install/python/lib/python3.13/ssl.py",
    "INFO: _tst_libssl: connecting to: www.python.org",
    "INFO: _tst_libssl: sock version: TLSv1.3",
    "INFO: _tst_datetime(): 'datetime' location = /home/jamie/tmp/influxdb3-pe/install/python/lib/python3.13/datetime.py",
    "INFO: _tst_datetime(): datetime.datetime.now() = 2025-03-14 15:19:39.727737",
    "INFO: _tst_datetime(): datetime.datetime.now().strftime() = 20250314151939"
  ],
  "database_writes": {},
  "errors": []
}

/home/jamie/tmp/influxdb3-pe/install/influxdb3 test wal_plugin -d mydb01 --lp="telegraf,cpu=cpu-total,host=foo usage_guest=0.0,usage_guest_nice=0.0,usage_idle=90.0,usage_iowait=0.3,usage_irq=0.0,usage_nice=0.0,usage_softirq=0.0,usage_steal=0.07,usage_system=0.7,usage_user=0.4" --input-arguments="host=foo" test-wal.py 
{
  "log_lines": [
    "INFO: _tst_query_basic(): query = 'SELECT * FROM telegraf LIMIT 1'",
    "INFO: _tst_query_basic(): result = [{'cpu': 'cpu-total', 'host': 'foo', 'time': 1741982645945594906, 'usage_guest': 0.0, 'usage_guest_nice': 0.0, 'usage_idle': 98.8476953907837, 'usage_iowait': 0.10020040080161435, 'usage_irq': 0.0, 'usage_nice': 0.0, 'usage_softirq': 0.0, 'usage_steal': 0.05010020040080273, 'usage_system': 0.6513026052104621, 'usage_user': 0.3507014028056636}]",
    "INFO: _tst_query_args(): args = '{'host': 'foo'}'",
    "INFO: _tst_query_args(): parameterized query = 'SELECT * FROM telegraf WHERE host = $host LIMIT 1'",
    "INFO: _tst_query_args(): parameterized query result = [{'cpu': 'cpu-total', 'host': 'foo', 'time': 1741982645945594906, 'usage_guest': 0.0, 'usage_guest_nice': 0.0, 'usage_idle': 98.8476953907837, 'usage_iowait': 0.10020040080161435, 'usage_irq': 0.0, 'usage_nice': 0.0, 'usage_softirq': 0.0, 'usage_steal': 0.05010020040080273, 'usage_system': 0.6513026052104621, 'usage_user': 0.3507014028056636}]",
    "INFO: _tst_table_batches(): table = telegraf",
    "INFO: _tst_table_batches(): row = {'cpu': 'cpu-total', 'host': 'foo', 'usage_guest': 0.0, 'usage_guest_nice': 0.0, 'usage_idle': 90.0, 'usage_iowait': 0.3, 'usage_irq': 0.0, 'usage_nice': 0.0, 'usage_softirq': 0.0, 'usage_steal': 0.07, 'usage_system': 0.7, 'usage_user': 0.4, 'time': 1741983753084091421}",
    "INFO: _tst_write_table(): line = test_table,tag1=tag1_value,tag2=tag2_value field1=1i,field2=2.0,field3=\"number three\"",
    "INFO: _tst_write_db(): line = other_table other_field=1i,other_field2=3.14 1741983753143361094"
  ],
  "database_writes": {
    "mydb01": [
      "test_table,tag1=tag1_value,tag2=tag2_value field1=1i,field2=2.0,field3=\"number three\""
    ],
    "mydb02": [
      "other_table other_field=1i,other_field2=3.14 1741983753143361094"
    ]
  },
  "errors": []
}

@jdstrand jdstrand deleted the jdstrand/use-python-standalone-3.13 branch August 15, 2025 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade python-build-standalone from 3.11 to 3.13

2 participants