Security concerns

[duckdoom4]

I have some concerns about the way some of this code is implemented.

To name the two I've noticed so far, the llm_math and sql_database chains.

It seems these two will blindly execute any code that is fed to it from the llm

This is a major security risk, since this opens anyone who uses these up for remote code execution. (The python one more then the sql one).

With a mitm attack, anyone can just return back a piece of code in the reply, pretending it is the bot. And if that's not enough, with some well crafted prompt, you can probably make it execute code as well (by making the llm return text with the same prompt pattern but custom python code)

I understand that this is in very early beta, but I've already seen this used in different places, due to ChatGPT's popularity.

In any case, it might be beneficial to switch from exec() to eval() for the python calculator, since eval() is build for the purpose of evaluating math expressions.

[duckdoom4]

https://blog.finxter.com/python-exec/
This should give a general idea of how the exec function can be abused

[phr33b4d]

I have a pull request in the adversarial-prompts repo of a prompt injection/code execution POC for Davinci. The scenarios duckdoom4 posits are bona fide threats. SQL injection is the number three attack on the most recent OWASP top ten. *See* https://owasp.org/www-project-top-ten/ (accessed Feb. 13, 2023). MITM attacks are also a common vulnerability. *See * https://owasp.org/www-community/attacks/Manipulator-in-the-middle_attack (accessed Feb. 13, 2023). Injection attacks (SQL and others) can be prevented via input sanitization*. See* https://security.berkeley.edu/education-awareness/how-protect-against-sql-injection-attacks (accessed Feb. 13, 2023). SQL Injection vulnerabilities can be prevented "in web applications by utilizing *parameterized database queries with bound, typed parameters and careful use of parameterized stored procedures in the database*". *Ibid*. (emphasis mine). As soon as possible, I intend to do a code review with these issues in mind. I will certainly do so before I utilize these features. I would encourage anyone else with infoSec experience to have a look, and feel free to reach out with any thoughts.

…

On Mon, Feb 13, 2023 at 3:46 PM duckdoom4 ***@***.***> wrote: I have some concerns about the way some of this code is implemented. To name the two I've noticed so far, the llm_math and sqldatabase chains. It seems these two will blindly execute any code that is fed to it from the llm. (I've not taken the time to fully delve into this, so please correct me if I'm wrong.) This is a major security risk, since this opens anyone who uses these up for remote code execution. (The python one more then the sql one). With a mitm attack, anyone can just return back a piece of code in the reply, pretending it is the bot. And if that's not enough, with some well crafted prompt, you can probably make it execute code as well. I understand that this is in very early beta, but I've already seen this used in different places, due to ChatGPT's popularity. — Reply to this email directly, view it on GitHub <#1026>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5NB4QG2BFYYXTPP7M6XM5TWXKTT7ANCNFSM6AAAAAAU2ZJSSA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[kjannette]

@duckdoom4 @hwchase17 please see hwchase17/adversarial-prompts#7 (referenced in post immediately above). There is a serious underlying vulnerability here: both in the models themselves and in add-on frameworks (i.e. Langchain) that open up another attack vector.

Containerization via Docker is a good start, for those willing and able to use it. See #1031

But it fails to address the underlying vulnerability. Also, Docker escapes are very possible, so that is at beast a band-aid. See https://attackdefense.pentesteracademy.com/listingnoauth?labtype=container-security-container-host-security&subtype=container-security-container-host-security-breakouts.

[blob42]

@kjannette I am putting together in #1031 the band-aid solution using a non priviliged docker user. I will follow up with a proposal update to the repls, bash and sql chains so they can execute in the container context when docker is setup.

I agree that the security risks are a big concern. My use of langchain will require heavy usage of shell contexts which I had to postpone until there is some minimal isolation.

Right now anyone pulling the repo and using the Repl/Shell/Sql chains is running arbitrary and potentially malicious code without any disclaimer.

May be adding a warning message that can be turned off by users so at least they are aware of the risks ?

[brandco]

Would running tools in docker containers solve some of the security issues?
It would be pretty easy to make a template out of a command like this
docker run --name python_langchain -it --rm python python -c "print({python_expression_to_be_evaluated})"

A docker tool might be a nice addition to langchain for other use cases as well.

[duckdoom4]

Would running tools in docker containers solve some of the security issues?
It would be pretty easy to make a template out of a command like this
docker run --name python_langchain -it --rm python python -c "print({python_expression_to_be_evaluated})"

A docker tool might be a nice addition to langchain for other use cases as well.

Well, it might help. But I don't think it's going to solve the core problem. Which is that {python_expression_to_be_evaluated} could contain unsafe code/ code with mal intent.

For example; Imagine someone supplying the following python code: )\nevil_python_function() or even worse, if the command line string is not sanitized: )"&[sys cmd to run]

As mentioned before, simply swapping from exec() to eval() would already help a great deal (But eval is still dangerous), tho depending on the final intentions might be too limiting.

Another option is having some tool that sanitizes the allowed python code with eg. regex patterns

[duckdoom4]

There are plenty of useful sources online on how to do it safely and what to look out for. Here are some examples:

https://stackoverflow.com/questions/10661079/restricting-pythons-syntax-to-execute-user-code-safely-is-this-a-safe-approach

PyPy sandbox (I recommend using this):
https://doc.pypy.org/en/latest/sandbox.html

Example of how difficult this problem is:
https://softwareengineering.stackexchange.com/questions/191623/best-practices-for-execution-of-untrusted-code/191628#191628

[blob42]

Quick update on the docker PR #1055 . I made a fully isolate container with no elevated privileges. The dependencies and the langchain package are all compiled in the image.

I avoided the virtualenv (poetry, pyenv, venv .. ) used a minimal python based image. When changed is detected only the the main package is installed.

Would be great to have some testers / feedback.

[duckdoom4]

@blob42 Check out PyPy's sandbox. This seems like a very well implemented sandboxing environment. Having that running in a docker container and we're basically as safe as can be. https://doc.pypy.org/en/latest/sandbox.html

[blob42]

@duckdoom4 thanks I will read about it and see how to integrate in the
container. I was thinking to have two separate images, one that can be
used for dev/testing with live reload (bind mount) and an other one especially crafted for agents.

[blob42]

I took a look to the sandboxlib project. I think should be feasible to use it for the Agent's container. However I noticed that there was no activity on the project since 3 years, the project's repo is here

There are other options that can be considered for running untrusted code. Here is a summary of what can be done on the sandboxing topic:

• Using a virtualized runtime for docker like gVisor
  Pros: Almost same sandboxing as full virtualization
  Cons: Performance ?

• Drop all capabilities from container see

• For Python: Use sandboxlib

• Update the exec family of Agent tools and allow execution on a remote shell (like ssh) . Users can redirect the shell to a full virtual machine (kvm, xen ... )

[duckdoom4]

I took a look to the sandboxlib project. I think should be feasible to use it for the Agent's container. However I noticed that there was no activity on the project since 3 years, the project's repo is here

There are other options that can be considered for running untrusted code. Here is a summary of what can be done on the sandboxing topic:

• Using a virtualized runtime for docker like gVisor
  Pros: Almost same sandboxing as full virtualization
  Cons: Performance ?

• Drop all capabilities from container see

• For Python: Use sandboxlib

• Update the exec family of Agent tools and allow execution on a remote shell (like ssh) . Users can redirect the shell to a full virtual machine (kvm, xen ... )

Ah yes, you clicked to link to the old repo. This one is up to date :): https://foss.heptapod.net/pypy/pypy

If you look at the 'how to' section on the page I mentioned, they posted that you should download from that link.

Tho RestrictedPython like used in #1134 might also be good enough

All the other things you mentioned also sound like a great idea 👍🏼

[kjannette]

Sandboxing will enclose a malicious actor in another perimeter once "in" - it definitely has value @duckdoom4. Sanitizing the input would aim to keep such an actor out in the first place. I have wanted to review the code and analyze vectors relating to this type of attack since it was first brought up, @blob42. Unfortunately, things at my org have been very busy and I have not had time. I am hoping that will change soon(ish) and I can submit a PR, if I draft one that closes a gap.

…

On Thu, Feb 16, 2023 at 4:26 PM duckdoom4 ***@***.***> wrote: @blob42 <https://github.com/blob42> Check out PyPy's sandbox. This seems like a very well implemented sandboxing environment. Having that running in a docker container and we're basically as safe as can be. https://doc.pypy.org/en/latest/sandbox.html — Reply to this email directly, view it on GitHub <#1026 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHE65U2KFFISTX7CQ57RRITWX2SQ3ANCNFSM6AAAAAAU2ZJSSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bborn]

I have a PR related to this that uses RestrictedPython (https://restrictedpython.readthedocs.io/):

#1134

Would love to get this (or something similar) merged.

[blob42]

I made some progress in #1266 . It is now possible to run arbitrary commands to any image or attach to existing containers. If gVisor is installed it will be automatically used. Check out the notebook for a few examples.

@bborn is there an official docker image for restrictedpython or something similar ? Is there a way to call the python process with a sane default sandbox environment with a simple command ? I could then make an image template (see docker/images.py) to use it by default.

[bborn]

@blob42 I'm not aware of anything like that. But yes, I think RestrictedPython is a good framework for setting up a sane sandbox with limited modules/functions.

Much of this is kind of out of my league (I'm just learning Python), but there is a lot that could be taken from here:

https://github.com/zopefoundation/AccessControl/tree/master/src/AccessControl

[manziman]

Been looking for a way to contribute now that I am using this awesome framework and I stumbled upon this thread.

Maybe someone has already addressed this - though I don't think I saw this suggested yet - but would os.chroot be a good (partial) mitigation for the code execution attack vector?

The concern I have with using Docker as the only option is that it limits the use of this particular tool to platforms to execution environments where you have access to Docker. For instance, if I wanted to run this in AWS Lambda or some other FaaS environment, docker isn't an option whereas chroot might be.

More generally, the best mitigation to this attack vector is to put in place defenses against MitM attacks more generally, which seems like something out of the scope of this framework, no?

Hoping there's somewhere I can contribute here.

[blob42]

@manziman I have been working on the docker API integration. Like you mentioned it is a very specific need so maybe it does not make sense to include it in langchain. I will post a link to the library when it's ready.

• FYI there is a project studying llm security here.

[manziman]

I have been hacking on this a little more in a related project. Needless to say, as many here probably already know, running untrusted code in python is practically speaking not possible without using something like docker or jails, which will complicate things. I am messing around with RestrictedPython but I can't really get it to work for my purposes.

Maybe a better approach would be using a domain specific language to serve the same purpose as the Python REPL util? Something like TextX? I've never used it before, maybe there's a better option as far as DSLs or metalanguages go for Python.

[bborn]

@manziman what issues are you having with RestrictedPython? I'm using it in my app and seems fine (granted I haven't tried to hard to break/test it). Here's what I'm doing:

https://gist.github.com/bborn/4c6e769e74f3d6397452bec3c9f294e6

[manziman]

@bborn

I am working on a way to abstract langchain behind an API, and store langchain objects in a SQL database. The idea is to have a deployable platform on which to run chains, using an API to allow other services to interact with those chains via API calls.

This works pretty well for standard chains, and sequential chains, as those can be represented pretty easily in JSON. Where things start to break down is in the Transform chains, as it requires you to pass in an actual python function.

So I was trying to see if I could allow the API caller to provide a string of python code which could be compiled to bytecode server-side for later execution (in a "secure" way).

[bborn]

@manziman me too! I've been building https://agent-hq.io - which sounds pretty similar. AgentHQ has a 'Custom' tool that lets users write python functions that can call the LLM and be used in Agents, but I use Restricted Python to try to avoid people rm -rfing my server. I know a fully isolated runtime (e.g. Docker or some serverless thing) is better, but this seems ok for now.

[manziman]

Very cool!

At this point for my use case I think I am going to solve the problem by allowing the user to provide a 'transform URL', which should be an external service serving POST requests containing the arguments to the function. That way you can host the 'transform logic' somewhere else and avoid running untrusted code in the main execution environment altogether.

The more I've been working on this the more I'm starting to think that storing functions as raw code or bytecode server side is asking for problems anyway, security considerations aside.

[mathetake]

Hi there from Wasm community, has anyone considered using Wasm as a sandboxed execution env for python repl? You can think of it as an isolated OS kernel in the userland (extremely lightweight) vs Docker containers sharing kernels with the host for example. See https://til.simonwillison.net/webassembly/python-in-a-wasm-sandbox for reference.

Let me know if it helps, and I would be more than happy to craft patches. Thanks!

[jcchavezs]

There is a similar concern in Coreruleset by running user scripts in the host (see https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-%28v3.x%29#user-content-exec) that we are most likely tackling in @corazawaf by using WASM.

[duckdoom4]

Hi there from Wasm community, has anyone considered using Wasm as a sandboxed execution env for python repl? You can think of it as an isolated OS kernel in the userland (extremely lightweight) vs Docker containers sharing kernels with the host for example. See https://til.simonwillison.net/webassembly/python-in-a-wasm-sandbox for reference.

Let me know if it helps, and I would be more than happy to craft patches. Thanks!

This seems like an awesome solution and something that can be implemented to work out of the box. That's great for the average user that needs to be protected, but doesn't need the extra protection a docker environment would provide.

[Jflick58]

@hwchase17 @dev2049 I’m planning on tackling a wasm compilation and execution solution to replace exec. For extra security, it may be worth dynamically scanning generated code and throwing errors if major vulnerabilities are flagged. At the least, it seems straightforward to me to replace exec() with wasm. I’m just checking that there’s no other work in flight on this. FYI #5294

[timxieICN]

Hi guys - any update on security improvement? Perhaps wasm compilation per #1026 (comment)

[aiakubovich]

any updates?

[ssutharzan]

Hi All, Does making a custom version of LangChain by removing code execution part break other functionalities of LangChain or create some other issues? I am thinking of using LangChain this way for now to minimixe code execution vulnerabilities. LangChain is well modularized and could be possible map the inter-code dependencies to customize and install only the needed parts per usecase to be extra safe and easy to review. In this way more mature/safe parts of LangChain can be started to be used in production systems.

[dosubot]

Hi, @duckdoom4! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you raised a concern about security issues with the llm_math and sql_database chains in the code. The suggestion was made to switch from exec() to eval() for the python calculator to mitigate this risk. There have been discussions about potential solutions such as using Docker containers, sandboxing with PyPy, RestrictedPython, and Wasm. Additionally, a pull request related to using RestrictedPython has been made.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution and please don't hesitate to reach out if you have any further questions or concerns!