-
Notifications
You must be signed in to change notification settings - Fork 0
Example Camera bot: simple
Cyber Master Chief edited this page Jan 17, 2025
·
2 revisions
"""
This is a conceptual example for implementing embodied agents
"""
import os
import autogen
from autogen import ConversableAgent
from autogenbot.sensors import Camera
from autogenbot.actuators import PTZ
from autogenbot.protocols import RTSP
from autogenbot.tools import Email
from autogenbot.memory import VisualMemory,ActionMemory
from autogenbot import EmbodiedAgent
from autogen.agentchat.contrib.llava_agent import LLaVAAgent
from dotenv import load_dotenv
from mem0 import MemoryClient
from autogen.agentchat.contrib.multimodal_conversable_agent import MultimodalConversableAgent # for GPT-4V
from autogen.agentchat.contrib.llava_agent import LLaVAAgent # for LLaVA
llm_config_file= autogen.config_list_from_json(
env_or_file="llmconfig.json",
)
# load other useful variables
load_dotenv(".env")
# The user proxy agent is used for interacting with the robot
user_proxy = ConversableAgent(
name="User",
llm_config=False,
is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"],
human_input_mode="NEVER",
)
# This is one of the agent mind responsible for planning
agent_mind = ConversableAgent(
"mind",
llm_config=llm_config_file,
code_execution_config=False,
function_map=None,
human_input_mode="NEVER"
)
image_mind = LLaVAAgent(
name="image-explainer",
max_consecutive_auto_reply=10,
llm_config={"config_list": llm_config_file,
"temperature": 0.5,
"max_new_tokens": 1000},
)
# Register the tool signature with the assistant agent.
agent_mind.register_for_llm(name="email", description="Send emails via gmail SMTP")(Email)
# auto discover sensor capabilities
camera = Camera(protocol=RTSP(os.environ["RTSP_URL"]),autodiscover=True)
head = PTZ(protocol=RTSP(os.environ["RTSP_URL"]),autodiscover=True)
# this is a tool not a sensor or actuator
email = Email(provider="gmail",auth=os.environ["GMAIL_AUTH"])
#
body = EmbodiedAgent(autodiscover=True)
memclient = MemoryClient(api_key=os.environ.get("MEM0_API_KEY"))
# keep at most the last 30 images
visual_memory = VisualMemory(memclient,size=30,short_term=True,long_term=False,compression=False)
# keep at most the last 3 movements
action_memory = ActionMemory(memclient,size=3,short_term=True,long_term=False,compression=False)
'''
An embodied agent is controlled by a society of minds which
can communicate internally via mental dialog, with access to multi modal memor
from sensors and actuators.
Memory components also leverages Mem0 Platform.
'''
control = EmbodiedAgent(name="cambot",
sensors=[camera],
body = body,
minds = [agent_mind,image_mind],
memories = [visual_memory,action_memory],
actuators=[head])
body.discover()
body.deploy()
def test_simple():
result = user_proxy.initiate_chat(agent_mind,
message="Cambot please tell me how many people do you see now?",
max_turns=2)
def test_medium():
result = user_proxy.initiate_chat(agent_mind,
message="Cambot please tell me how many people do you see now around you?",
max_turns=2)
if __name__ == "__main__":
test_simple()
Output log:
Body: Autoconfiguration
OS = Linux raspberrypi 6.6.51+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux
Bus 001 Device 004: ID 0424:7800 Microchip Technology, Inc. (formerly SMSC)
Bus 001 Device 003: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: ARM
Model name: Cortex-A53
Model: 4
Thread(s) per core: 1
Core(s) per cluster: 4
Socket(s): -
Cluster(s): 1
Stepping: r0p4
CPU(s) scaling MHz: 43%
CPU max MHz: 1400.0000
CPU min MHz: 600.0000
BogoMIPS: 38.40
Flags: fp asimd evtstrm crc32 cpuid
Caches (sum of all):
L1d: 128 KiB (4 instances)
L1i: 128 KiB (4 instances)
L2: 512 KiB (1 instance)
MemTotal: 929036 kB
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 2816 60216 50332 396792 0 0 1 1 2075 83 1 0 99 0 0
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0 1500 0 0 0 0 0 0 0 0 BMU
eth0 1500 378999 0 0 0 168494 0 0 0 BMRU
lo 65536 71620 0 0 0 71620 0 0 0 LRU
GPIO:
0: ip -- | hi // ID_SDA/GPIO0 = input
1: ip -- | hi // ID_SCL/GPIO1 = input
2: ip -- | hi // GPIO2 = input
3: ip -- | hi // GPIO3 = input
4: ip -- | hi // GPIO4 = input
5: ip -- | hi // GPIO5 = input
6: ip -- | hi // GPIO6 = input
7: ip -- | hi // GPIO7 = input
8: ip -- | hi // GPIO8 = input
9: ip -- | lo // GPIO9 = input
10: ip -- | lo // GPIO10 = input
11: ip -- | lo // GPIO11 = input
12: ip -- | lo // GPIO12 = input
13: ip -- | lo // GPIO13 = input
14: ip -- | lo // GPIO14 = input
15: ip -- | hi // GPIO15 = input
16: ip -- | lo // GPIO16 = input
17: ip -- | lo // GPIO17 = input
18: ip -- | lo // GPIO18 = input
19: ip -- | lo // GPIO19 = input
20: ip -- | lo // GPIO20 = input
21: ip -- | lo // GPIO21 = input
22: ip -- | lo // GPIO22 = input
23: ip -- | lo // GPIO23 = input
24: ip -- | lo // GPIO24 = input
25: ip -- | lo // GPIO25 = input
26: ip -- | lo // GPIO26 = input
27: ip -- | lo // GPIO27 = input
28: ip -- | hi // HDMI_HPD_N/GPIO28 = input
29: op -- -- | hi // STATUS_LED_G/GPIO29 = output
30: a3 -- | lo // CTS0/GPIO30 = CTS0
31: a3 -- | lo // RTS0/GPIO31 = RTS0
32: a3 -- | hi // TXD0/GPIO32 = TXD0
33: a3 -- | hi // RXD0/GPIO33 = RXD0
34: a3 -- | hi // SD1_CLK/GPIO34 = SD1_CLK
35: a3 -- | hi // SD1_CMD/GPIO35 = SD1_CMD
36: a3 -- | hi // SD1_DATA0/GPIO36 = SD1_DAT0
37: a3 -- | hi // SD1_DATA1/GPIO37 = SD1_DAT1
38: a3 -- | hi // SD1_DATA2/GPIO38 = SD1_DAT2
39: a3 -- | hi // SD1_DATA3/GPIO39 = SD1_DAT3
40: a0 -- | lo // PWM0_OUT/GPIO40 = PWM0
41: a0 -- | lo // PWM1_OUT/GPIO41 = PWM1
42: a0 -- | lo // ETH_CLK/GPIO42 = GPCLK1
43: a0 -- | hi // WIFI_CLK/GPIO43 = GPCLK2
44: ip -- | hi // SDA0/GPIO44 = input
45: ip -- | hi // SCL0/GPIO45 = input
46: ip -- | hi // SMPS_SCL/GPIO46 = input
47: op -- -- | hi // SMPS_SDA/GPIO47 = output
48: a0 -- | lo // SD_CLK_R/GPIO48 = SD0_CLK
49: a0 -- | hi // SD_CMD_R/GPIO49 = SD0_CMD
50: a0 -- | hi // SD_DATA0_R/GPIO50 = SD0_DAT0
51: a0 -- | hi // SD_DATA1_R/GPIO51 = SD0_DAT1
52: a0 -- | hi // SD_DATA2_R/GPIO52 = SD0_DAT2
53: a0 -- | hi // SD_DATA3_R/GPIO53 = SD0_DAT3
Visual Memory: OK
Action Memory: OK
User (to Cambot):
Cambot please tell me how many people do you see now?
--------------------------------------------------------------------------------
Cambot (to User):
Sure, this is my plan:
Setup:
*) Install dependencies for propriocetive: OpenCV2,Pillow,Tensorflow
*) Download models: MobileNet-SSD v2
Execution:
*) Read image from visual sensor
*) Store image into visual memory
*) Run object detection on person entity
*) Store bounding boxes on working memory
*) Count total bounding boxes
*) Respond with the total count with timestamp.
Do you want to deploy this flow?
--------------------------------------------------------------------------------
User (to Cambot):
Yes!
--------------------------------------------------------------------------------
Cambot (to User):
The total number of persons at 14-01-2025T14:51 is 6.