Example Walkthrough

Example Walkthrough

How the YouTube example works, step by step. Every API mentioned is documented in docs/lua-api.md (Traditional Chinese). Code comments in the example are in Chinese; the snippets below are excerpts.

Opening it in VS Code

The example is already on the phone once you've added it (Run the Example). After connecting VS Code (Your First Script), it syncs into your local folder as youtube-rickroll/: edit it and press F5 like any other script.

script.json: its own display

"display": { "width": 720, "height": 1280, "densityDpi": 320, "keyboard": false }

The script gets its own 720×1280 virtual display. keyboard: false stops the phone's input method from attaching to apps on that display, so the keys input.text sends arrive as typed instead of being composed by, say, a Chinese keyboard (Android 12 and later).

The code never hardcodes coordinates: everything comes from screen.width, screen.height, or a match result, so it works on other display sizes too.

1. Open search

app.launch("com.google.android.youtube")
input.key(KEYCODE_SEARCH)
local _, hit = vision.wait_any({
    { text = "搜尋", roi = search_bar },
    { text = "Search", roi = search_bar },
}, 3000)

• app.launch starts YouTube on the script's display.
• YouTube may resume on any page, so instead of looking for a search button the script sends the search key (input.key), which opens search everywhere. A cold start can swallow it, so it retries up to five times.
• vision.wait_any waits for whichever request matches first. Each is a text request, so OCR looks for the word in either language. roi limits recognition to the top-left of the screen: full-screen OCR is slow, and the right half may be covered by a mini player.

2. Type the query

input.text("rick astley never gonna give you up\n")

input.text types ASCII into the focused field; the trailing \n presses Enter.

3. Find the video

video = vision.wait({ text = "Rick Astley - Never Gonna", roi = below_search }, 10000, 1000)
input.swipe({ W // 2, H * 3 // 4, W // 2, H // 4 }, 400)
input.tap(video.cx, video.cy)

• Ads often top the results, so the script looks for the official video's title (artist - title) rather than tapping the first result.
• The third argument of vision.wait is step_ms: run recognition at most once a second instead of on every frame, since OCR is expensive.
• If the title isn't visible, input.swipe scrolls down and it tries again.
• A match has cx, cy at the centre of the matched text, ready for input.tap.

4. Skip ads

local AD_SIGNS = {
    { text = "略過", exact = true, roi = PLAYER },
    { text = "Skip ad", exact = true, roi = PLAYER },
    -- …
    { text = "Sponsored", roi = PLAYER },
}
local idx, hit = vision.find_any(AD_SIGNS)

vision.find_any checks the latest frame once and returns the index of the first match. The skip buttons come first in the list, so a match there means tap it; a match on the "Sponsored" label means an ad is still playing. exact = true matches the whole line, so the countdown text "you can skip in 5 seconds" isn't mistaken for the button. The script waits until three quiet seconds pass with no ad.

5. Read the time

for _, line in ipairs(vision.read_lines({ 0, PLAYER_H // 2, W, PLAYER_H // 2 })) do
    local now, total = line.text:match("(%d+:%d+)%s*/%s*(%d+:%d+)")

After tapping the player to show its controls, vision.read_lines returns every line of text in the lower half of the player, and a Lua pattern picks out 0:12 / 3:33. A total under a minute means it read an ad's clock, so it's ignored.

6. Drag the progress bar

input.down(1, from_x, y)
for step = 1, 10 do
    sleep(30)
    input.move(1, from_x + (to_x - from_x) * step // 10, y)
end
input.up(1)

input.down, input.move and input.up drive one finger (pointer 1) step by step, which a single swipe can't do. YouTube's scrubbing is relative, so the finger starts at the current position and moves by the distance to the target.

7. Confirm and record

local reading = vision.read({ label.x - 8, label.y - 4, label.w + 40, label.h + 8 })
data.set("position", result)
data.set("status", "never gonna give you up")

vision.read treats a small box as a single line and recognizes it directly, which is faster and more accurate than detecting lines first. The box is the time label found in step 5, widened because the text gets longer. If the time is within 10 seconds of the target the script records it with data.set, which shows up under Published Script Data in the app and in VS Code. Ads can interrupt steps 4–7 at any point, so the whole sequence retries up to three times.

Failing clearly

local function fail(reason)
    data.set("status", reason)
    device.notify("YouTube 範例", reason)
    error(reason)
end

Every give-up point calls fail: it records the reason, posts a notification with device.notify, and stops the script with error, which makes the app show Error: with the reason.