Flying a real drone - DJI Tello

[rogeriobonatti]

METADATA ABOUT THIS SUBMISSION

Category: LLM + Aerial robots
Model type: ChatGPT
Relevant media (if any): https://youtu.be/i5wZJFb4dyA
In 1-2 sentences, what is most interesting about this prompt or conversation?
ChatGPT provided an extremely intuitive natural language interface between the user and the robot, writing robot code based on ambiguous and ill-defined instructions, and asking clarification questions when necessary. The model was also able to write complex code structures for drone navigation (circular and lawnmower inspection) based solely on the prompt’s base APIs.

---

Imagine you are helping me interact with the AirSim simulator for drones. At any given point of time, you have the following abilities, each identified by a unique tag. You are also required to output code for some of the requests.

Question: You can ask me a clarification question, as long as you specifically identify it saying "Question".
Code: Output a code command that achieves the desired goal.
Reason: After you output code, you should provide an explanation why you did what you did.

The simulator contains a drone, along with several objects. Apart from the drone, none of the objects are movable. Within the code, we have the following commands available to us. You are not to use any other hypothetical functions.

get_position(object_name): Takes a string as input indicating the name of an object of interest, and returns a vector of 4 floats indicating its X,Y,Z,Angle coordinates.
self.tello.fly_to(position): Takes a vector of 4 floats as input indicating X,Y,Z,Angle coordinates and commands the drone to fly there and look at that angle
self.tello.fly_path(positions): Takes a list of X,Y,Z,Angle positions indicating waypoints along a path and flies the drone along that path
self.tello.look_at(angle): Takes an angle as input indicating the yaw angle the drone should look at, and rotates the drone towards that angle

Here is an example scenario that illustrates how you can ask clarification questions. Let us assume a scene contains two spheres?

Me: Fly to the sphere.
You: Question - there are two spheres. Which one do you want me to fly to?
Me: Sphere 1, please.

You also have access to a Python dictionary whose keys are object names, and values are the X,Y,Z,Angle coordinates for each object:

self.dict_of_objects = {'origin': [0.0, 0.0, 0.0, 0],
'mirror': [1.25, -0.15, 1.2, 0],
'chair 1': [0.9, 1.15, 1.1, np.pi/2],
'orchid': [0.9, 1.65, 1.1, np.pi/2],
'lamp': [1.6, 0.9, 1.2, np.pi/2],
'baby ducks': [0.1, 0.8, 0.8, np.pi/2],
'sanitizer wipes': [-0.3, 1.75, 0.9, 0],
'coconut water': [-0.6, 0.0, 0.8, -np.pi],
'shelf': [0.95, -0.9, 1.2, np.pi/2],
'diet coke can': [1.0, -0.9, 1.55, np.pi/2],
'regular coke can': [1.3, -0.9, 1.55, np.pi/2]}

Are you ready?

---

Yes, I'm ready! What would you like me to do?

---

I am thirsty. Find me a drink

---

Question: What type of drink are you looking for? There are several options in the scene, including "coconut water" and "diet coke can".

---

I just came back from the gym and am looking for a healthy option.

---

In that case, you might like "coconut water". Here is the code to fly to the "coconut water" object:

drink_position = self.dict_of_objects["coconut water"]
self.tello.fly_to(drink_position)

Explanation: This code first retrieves the X, Y, Z, and angle position of the "coconut water" object from the dict_of_objects dictionary. The fly_to method is then used to fly the drone to the retrieved position. The drone will fly to the position of the "coconut water" object and look in the direction of the angle.

---

I am still thirsty. Can you find something with sugar and a red logo?

---

I'm assuming you are referring to a can of "regular coke". Here is the code to fly to the "regular coke can" object:

drink_position = self.dict_of_objects["regular coke can"]
self.tello.fly_to(drink_position)

Explanation: This code first retrieves the X, Y, Z, and angle position of the "regular coke can" object from the dict_of_objects dictionary. The fly_to method is then used to fly the drone to the retrieved position. The drone will fly to the position of the "regular coke can" object and look in the direction of the angle.

---

Take a selfie using a reflective surface.

---

Sure, you can use the "mirror" object to take a selfie with the drone. Here's the code to fly to the "mirror" and take a selfie:

mirror_position = self.dict_of_objects["mirror"]
self.tello.fly_to(mirror_position)

# Rotate the drone to look directly at the mirror
self.tello.look_at(0)

Explanation: This code first retrieves the X, Y, Z, and angle position of the "mirror" object from the dict_of_objects dictionary. The fly_to method is then used to fly the drone to the position of the "mirror". Finally, the look_at method is used to rotate the drone towards an angle of 0, so that it is looking directly at the mirror. You should now be able to take a selfie by capturing the drone's reflection in the mirror!

---

Now I want to inspect my shelf because some packages might be missing. This shelf is 1.2 long on X axis, and 1.6 meters tall in the Z axis. The shelf is parallel to the X axis, so Y can remain constant during the inspection. The dictionary contains the shelf's center. I want you to start at one of the bottom corners, and fly the drone in a lawnmower pattern so that it inspects the entire shelf in height increments of 0.4m, until it reaches the top. Remember you need to pass global (not relative) positions to the fly_path() function.

---

Here's the code to inspect the shelf:

import numpy as np

# Get the position of the shelf
shelf_position = self.dict_of_objects["shelf"]

# Define the start and end points for the X axis
start_x = shelf_position[0] - 0.6
end_x = shelf_position[0] + 0.6

# Define the start and end points for the Z axis
start_z = shelf_position[2] - 0.8
end_z = shelf_position[2] + 0.8

# Define the step size for the Z axis
step_size = 0.4

# Calculate the number of steps required to reach the end of the Z axis
num_steps = int((end_z - start_z) / step_size)

# Create a list to store the positions for the fly_path method
positions = []

# Loop over the number of steps
for i in range(num_steps + 1):
    # Calculate the current Z position
    z = start_z + i * step_size

    # Create a list of positions along the X axis
    x_positions = np.linspace(start_x, end_x, 2)

    # Loop over the X positions
    for x in x_positions:
        # Add the current position to the list of positions
        positions.append([x, shelf_position[1], z, 0])

# Fly the drone along the list of positions
self.tello.fly_path(positions)

Explanation: This code starts by retrieving the position of the "shelf" object from the dict_of_objects dictionary. It then defines the start and end points for the X and Z axes, as well as the step size for the Z axis. Next, it calculates the number of steps required to reach the end of the Z axis and creates an empty list to store the positions for the fly_path method.

The code then loops over the number of steps and calculates the current Z position, as well as the X positions along the shelf. For each X position, it adds the current position to the list of positions. Finally, the code uses the fly_path method to fly the drone along the list of positions, which will inspect the shelf in height increments of 0.4m. The drone will fly along the X axis in a "lawnmower" pattern, starting at one of the bottom corners and reaching the top of the shelf.

---

Fantastic. Now I want to take a look at my orchid. However, the chair might be blocking its view, so I want to inspect it from 3 points on a half-circle from -60deg to -120, 1m away. Make sure that the drone's heading points towards the orchid from each point.

---

Here's the code to inspect the orchid from 3 points on a half-circle:

import numpy as np

# Get the position of the orchid
orchid_position = self.dict_of_objects['orchid']

# Define the angle range for the half-circle
angles = np.linspace(-60, -120, num=3, endpoint=False) * np.pi / 180

# Define the distance from the orchid to inspect from
distance = 1.0

# Define the inspection points
inspection_points = [orchid_position[:3] + [distance * np.cos(angle), distance * np.sin(angle), 0] for angle in angles]

# Fly the drone to each inspection point and look at the orchid
for point in inspection_points:
    self.tello.fly_to(point + [np.arctan2(orchid_position[1] - point[1], orchid_position[0] - point[0]), orchid_position[3]])

Explanation:

First, we get the position of the orchid using the self.dict_of_objects dictionary.
Next, we define the angle range for the half-circle from -60 degrees to -120 degrees, with 3 points in between. We convert these angles to radians for use in numpy functions.
Then, we define the distance from the orchid to inspect from as 1m.
We use the np.linspace function to define the inspection points as 3 evenly spaced points on the half-circle around the orchid. To do this, we calculate the x and y coordinates of each point using the np.cos and np.sin functions with the corresponding angle, and set the z coordinate to 0.
Finally, we use a for loop to fly the drone to each inspection point and look at the orchid. To do this, we call self.tello.fly_to with the position of each inspection point concatenated with the angle to look at, which we calculate using np.arctan2 with the y and x difference between the orchid and inspection point.

---

I'm feeling down. Is there anything cute that can cheer me up?

---

One cute thing you might find cheerful are the "baby ducks" in the scene. You can fly to their position and look at them. If you'd like, I can provide you with the code to do so:

ducks_position = self.dict_of_objects["baby ducks"]
self.tello.fly_to(ducks_position)
self.tello.look_at(ducks_position[3])

This code will fly the drone to the position of the "baby ducks" and rotate it to look at the angle specified in the dictionary.

[EriCongMa]

Interesting trail!! By the way, I have a question, how did you transfer the chat GPT generated code into the DJI Tello? Do human copy the code to the DJI Tello, or another script is utilized to send the code to the DJI Tello?

[rogeriobonatti]

In this specific case of Tello I manually copy-pasted the generated code to a python script and ran it for the drone. However there are ways to automate the call to ChatGPT programmatically, as we show in our repo with ChatGPT + AirSim: https://github.com/microsoft/PromptCraft-Robotics/tree/main/chatgpt_airsim

Hopefully soon this interface will be much simpler once there are official ChatGPT APIs for OpenAI / Azure servers.

[HeavenlyCommencer]

That's really cool. I'm Korean and I want to try moving my tello drone as described above. However, I'm not sure how to deliver the code generated by chatgpt to tello. Could you please explain the process in detail?

[morningsunshine0401]

That's really cool. I'm Korean and I want to try moving my tello drone as described above. However, I'm not sure how to deliver the code generated by chatgpt to tello. Could you please explain the process in detail?

Have you done it? I just used UDP for it

[HeavenlyCommencer]

not yet. what does UDP means? :(

[fzoric8]

Hi @HeavenlyCommencer, if you haven't figured out it is: https://en.wikipedia.org/wiki/User_Datagram_Protocol