from tiktoken import _tiktoken is causing circular dependency on AWS lambda

[akshat-g]

Error:
cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import)

I have added all python packages as a AWS layer and my lambda functions are accessing those dependencies via AWS layer.
There is no file named tiktoken.py in my project.

_tiktoken is defined in a rust file. Is there anything I am missing while packaging tiktoken for AWS lambda?
I need help here. Have been stuck in this for a while now.

[hauntsaninja]

I don't know much about lambda or what an AWS layer is. How are you building / installing tiktoken?

[alvarobartt]

Can you share a file with the Python dependencies you have installed in that environment? i.e. pip list >> dependencies.txt

Asking because in the past some dependencies were running into conflicts with tiktoken, so this may be the case here too.

[btakeya]

same issue on aws glue using tiktoken-0.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl and vanilla build (tag 0.5.1 -- 39f29ce) on my laptop.

[dbmanifest]

any fix for this?

[hauntsaninja]

There's little I can do without a clear repro. If you're having this issue, I recommend:

• post the full stacktrace
• post where Python's import system resolves tiktoken and tiktoken._tiktoken, i.e. the results of importlib.util.find_spec("tiktoken._tiktoken") and importlib.util.find_spec("tiktoken")

[mrdomino]

I may have a clean repro. I'm trying to make a Portfile for MacPorts. PR is macports/macports-ports#22094.

If you clone that and apply the following patch:

--- Portfile.old	2024-01-08 21:29:33
+++ Portfile	2024-01-08 21:31:50
@@ -36,6 +36,9 @@
                     port:py${python.version}-regex \
                     port:py${python.version}-requests

+    depends_test-append \
+                    port:py${python.version}-hypothesis
+
     # cd ${worksrcpath}
     # sudo cargo update
     # egrep -e '^(name|version|checksum) = ' Cargo.lock | perl -pe 's/^(?:name|version|checksum) = "(.+)"/$1/' | tr '\n' ' ' | perl -pe 's|([0-9a-f]{64})|\1 \\\n|g' | pbcopy
@@ -86,5 +89,7 @@
                     windows_x86_64_gnullvm 0.48.5 0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc \
                     windows_x86_64_msvc 0.48.5 ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538

+    test.run        yes
+
     livecheck.type  none
 }

And then run sudo port test py312-tiktoken, the error will show up.

[mrdomino]

In case it's helpful, here's the relevant excerpt of the log of running that command:

test.log

Stack trace:

:debug:test system:  cd "/opt/local/var/macports/build/_Users_joshin_macports-ports_python_py-tiktoken/py312-tiktoken/work/tiktoken-0.5.2" && py.test-3.12 -o addopts='' 
:info:test ================================== test session starts ===================================
:info:test platform darwin -- Python 3.12.1, pytest-7.4.3, pluggy-1.3.0
:info:test rootdir: /opt/local/var/macports/build/_Users_joshin_macports-ports_python_py-tiktoken/py312-tiktoken/work/tiktoken-0.5.2
:info:test plugins: hypothesis-6.92.1
:info:test collected 0 items / 5 errors
:info:test ========================================= ERRORS =========================================
:info:test ________________________ ERROR collecting tests/test_encoding.py _________________________
:info:test ImportError while importing test module '/opt/local/var/macports/build/_Users_joshin_macports-ports_python_py-tiktoken/py312-tiktoken/work/tiktoken-0.5.2/tests/test_encoding.py'.
:info:test Hint: make sure your test modules/packages have valid Python names.
:info:test Traceback:
:info:test /opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
:info:test     return _bootstrap._gcd_import(name[level:], package, level)
:info:test tests/test_encoding.py:9: in <module>
:info:test     import tiktoken
:info:test tiktoken/__init__.py:2: in <module>
:info:test     from .core import Encoding as Encoding
:info:test tiktoken/core.py:9: in <module>
:info:test     from tiktoken import _tiktoken
:info:test E   ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import) (/opt/local/var/macports/build/_Users_joshin_macports-ports_python_py-tiktoken/py312-tiktoken/work/tiktoken-0.5.2/tiktoken/__init__.py)

[hauntsaninja]

If you're repackaging this it's pretty clear how it would show up, e.g. when testing you may need to specify import mode like so: https://github.com/openai/tiktoken/blob/main/pyproject.toml#L40 This makes sense given how pytest works: https://docs.pytest.org/en/7.1.x/explanation/pythonpath.html

But I don't use AWS Lambda, so I'm not sure how those folks are encountering this. Presumably they're not trying to pytest tiktoken.

[mrdomino]

You're right - I had managed to set up PYTHONPATH and to set --import-mode=append but never at the same time. Thanks.

[hauntsaninja]

Closing, since it's been several months without a clear repro. If you do have a reproducer, please post the information I requested in #184 (comment)