diff --git a/assets/tutorials/data-management/tensor-output.png b/assets/tutorials/data-management/tensor-output.png new file mode 100644 index 0000000000..246814efe4 Binary files /dev/null and b/assets/tutorials/data-management/tensor-output.png differ diff --git a/docs/data-ai/ai/run-inference.md b/docs/data-ai/ai/run-inference.md index 92cdbd5068..a5e9a047a8 100644 --- a/docs/data-ai/ai/run-inference.md +++ b/docs/data-ai/ai/run-inference.md @@ -15,100 +15,113 @@ aliases: - /ml/vision/segmentation/ - /ml/vision/ - /get-started/quickstarts/detect-people/ -description: "Run inference on a model with a vision service or an SDK." +description: "Run machine learning inference locally on your robot or remotely in the cloud using vision services, ML model services, or SDKs." +date: "2025-09-12" --- Inference is the process of generating output from a machine learning (ML) model. -With Viam, you can run inference to generate the following kinds of output: -- object detection (using bounding boxes) -- classification (using tags) - -You can run inference locally on a Viam machine, or remotely in the Viam cloud. +You can run inference [locally on a Viam machine](#machine-inference), or [remotely in the Viam cloud](#cloud-inference). ## Machine inference -You can use `viam-server` to deploy and run ML models directly on your machines. - -You can run inference on your machine in the following ways: - -- with a vision service -- manually in application logic with an SDK +When you have [deployed an ML model](/data-ai/ai/deploy/) on your machine, you can run inference on your machine [directly with the ML model service](#using-an-ml-model-service-directly) or [using a vision service](#using-a-vision-service) that interprets the inferences. Entry-level devices such as the Raspberry Pi 4 can run small ML models, such as TensorFlow Lite (TFLite). -More powerful hardware, including the Jetson Xavier or Raspberry Pi 5 with an AI HAT+, can process larger AI models, including TensorFlow and ONNX. +More powerful hardware, including the Jetson Xavier or Raspberry Pi 5 with an AI HAT+, can process larger models, including TensorFlow and ONNX. +If your hardware does not support the model you want to run, see [Cloud inference](#cloud-inference). + +### Using an ML model service directly {{< tabs >}} -{{% tab name="Vision service" %}} +{{% tab name="Web UI" %}} -Vision services apply an ML model to a stream of images from a camera to generate bounding boxes or classifications. +1. Visit your machine's **CONFIGURE** or **CONTROL** page. +1. Expand the **TEST** area of the ML model service panel to view the tensor output. -{{}} +{{< imgproc src="/tutorials/data-management/tensor-output.png" alt="Example tensor output" resize="x1000" class="shadow imgzoom fill" >}} + +{{% /tab %}} +{{% tab name="Python" %}} + +The following code passes an image to an ML model service, and uses the [`Infer`](/dev/reference/apis/services/ml/#infer) method to make inferences: + +{{< read-code-snippet file="/static/include/examples-generated/run-inference.snippet.run-inference.py" lang="py" class="line-numbers linkable-line-numbers" data-line="82-85" >}} + +{{% /tab %}} +{{% tab name="Go" %}} + +The following code passes an image to an ML model service, and uses the [`Infer`](/dev/reference/apis/services/ml/#infer) method to make inferences: + +{{< read-code-snippet file="/static/include/examples-generated/run-inference.snippet.run-inference.go" lang="go" class="line-numbers linkable-line-numbers" data-line="166-171" >}} + +{{% /tab %}} +{{< /tabs >}} + +### Using a vision service + +There are a range of vision services that interpret images. +Some vision services apply an ML model to a stream of images from a camera to: -{{% alert title="Tip" color="tip" %}} -Some vision services include their own ML models, and thus do not require a deployed ML model. -If your vision service does not include an ML model, you must [deploy an ML model to your machine](/data-ai/ai/deploy/) to use that service. -{{% /alert %}} +- detect objects (using bounding boxes) +- classify (using tags) To use a vision service: -1. Visit your machine's **CONFIGURE** page. +1. Navigate to your machine's **CONFIGURE** page. 1. Click the **+** icon next to your main machine part and select **Component or service**. -1. Type in the name of the service and select a vision service. -1. If your vision service does not include an ML model, [deploy an ML model to your machine](/data-ai/ai/deploy/) to use that service. -1. Configure the service based on your use case. -1. To view the deployed vision service, use the live detection feed. - The feed shows an overlay of detected objects or classifications on top of a live camera feed. - On the **CONFIGURE** or **CONTROL** pages for your machine, expand the **Test** area of the service panel to view the feed. +1. Select a vision service. + For more information on the vision service, see its entry in the [Viam registry](https://app.viam.com/registry). +1. Configure your vision service. + If your vision service does not include an ML model, you need to [deploy an ML model to your machine](/data-ai/ai/deploy/) and select it when configuring your vision service. - {{< imgproc src="/tutorials/data-management/blue-star.png" alt="Detected blue star" resize="x200" class="shadow" >}} - {{< imgproc src="/tutorials/filtered-camera-module/viam-figure-preview.png" alt="Detection of a viam figure with a confidence score of 0.97" resize="x200" class="shadow" >}} +{{% expand "Click to search available vision services" %}} -For instance, you could use [`viam:vision:mlmodel`](/operate/reference/services/vision/mlmodel/) with the `EfficientDet-COCO` ML model to detect a variety of objects, including people, bicycles, and apples, in a camera feed. - -Alternatively, you could use [`viam-soleng:vision:openalpr`](https://app.viam.com/module/viam-soleng/viamalpr) to detect license plates in images. -Since this service includes its own ML model, there is no need to configure a separate ML model. +{{}} -After adding a vision service, you can use a vision service API method with a classifier or a detector to get inferences programmatically. -For more information, see the APIs for [ML Model](/dev/reference/apis/services/ml/) and [Vision](/dev/reference/apis/services/vision/). +{{% /expand%}} -{{% /tab %}} -{{% tab name="SDK" %}} +{{% expand "Click to view example vision services" %}} -With the Viam SDK, you can pass image data to an ML model service, read the output annotations, and react to output in your own code. -Use the [`Infer`](/dev/reference/apis/services/ml/#infer) method of the ML Model API to make inferences. + +| Example | Description | +| ------- | ----------- | +| Detect a variety of objects | Use the [`viam:vision:mlmodel`](/operate/reference/services/vision/mlmodel/) vision service with the `EfficientDet-COCO` ML model to detect a variety of objects, including people, bicycles, and apples, in a camera feed. | +| Detect license plates | Use the [`viam-soleng:vision:openalpr`](https://app.viam.com/module/viam-soleng/viamalpr) vision service to detect license plates in images. This service includes its own ML model. | -For example: +{{% /expand%}} {{< tabs >}} -{{% tab name="Python" %}} +{{% tab name="Web UI" %}} -```python {class="line-numbers linkable-line-numbers"} -import numpy as np +1. Visit your machine's **CONFIGURE** or **CONTROL** page. +1. Expand the **TEST** area of the vision service panel. -my_mlmodel = MLModelClient.from_robot(robot=machine, name="my_mlmodel_service") + The feed shows an overlay of detected objects or classifications on top of a live camera feed. + + {{< imgproc src="/tutorials/data-management/blue-star.png" alt="Detected blue star" resize="x200" class="shadow" >}} + {{< imgproc src="/tutorials/filtered-camera-module/viam-figure-preview.png" alt="Detection of a viam figure with a confidence score of 0.97" resize="x200" class="shadow" >}} -image_data = np.zeros((1, 384, 384, 3), dtype=np.uint8) +{{% /tab %}} +{{% tab name="Python" %}} -# Create the input tensors dictionary -input_tensors = { - "image": image_data -} +The following code passes an image from a camera to a vision service and uses the [`GetClassifications`](/dev/reference/apis/services/vision/#getclassifications) method: -output_tensors = await my_mlmodel.infer(input_tensors) -``` +{{< read-code-snippet file="/static/include/examples-generated/run-vision-service.snippet.run-vision-service.py" lang="py" class="line-numbers linkable-line-numbers" data-line="36-37" >}} {{% /tab %}} {{% tab name="Go" %}} -```go {class="line-numbers linkable-line-numbers"} -input_tensors := ml.Tensors{"0": tensor.New(tensor.WithShape(1, 2, 3), tensor.WithBacking([]int{1, 2, 3, 4, 5, 6}))} +The following code passes an image from a camera to a vision service and uses the [`GetClassifications`](/dev/reference/apis/services/vision/#getclassifications) method: -output_tensors, err := myMLModel.Infer(context.Background(), input_tensors) -``` +{{< read-code-snippet file="/static/include/examples-generated/run-vision-service.snippet.run-vision-service.go" lang="go" class="line-numbers linkable-line-numbers" data-line="66-69" >}} {{% /tab %}} -{{< /tabs >}} +{{% tab name="TypeScript" %}} + +The following code passes an image from a camera to a vision service and uses the [`GetClassifications`](/dev/reference/apis/services/vision/#getclassifications) method: + +{{< read-code-snippet file="/static/include/examples-generated/run-vision-service.snippet.run-vision-service.ts" lang="ts" class="line-numbers linkable-line-numbers" data-line="32-38" >}} {{% /tab %}} {{< /tabs >}} @@ -116,19 +129,17 @@ output_tensors, err := myMLModel.Infer(context.Background(), input_tensors) ## Cloud inference Cloud inference enables you to run machine learning models in the Viam cloud, instead of on a local machine. -Cloud inference often provides more computing power than edge devices, so you can benefit from: +Cloud inference provides more computing power than edge devices, enabling you to run more computationally-intensive models or achieve faster inference times. -- larger, more accurate models -- faster inference times +You can run cloud inference using any TensorFlow and TensorFlow Lite model in the Viam registry, including unlisted models owned by or shared with you. -You can run cloud inference using any TensorFlow and TensorFlow Lite model in the Viam registry, including private models owned by or shared with your organization. - -To run cloud inference, you must pass +To run cloud inference, you must pass the following: - the binary data ID and organization of the data you want to run inference on - the name, version, and organization of the model you want to use for inference -The [`viam infer`](/dev/tools/cli/#infer) CLI command runs inference in the cloud on a piece of data using the specified ML model: +You can obtain the binary data ID from the [**DATA** tab](https://app.viam.com/data/view) and the organization ID by running the CLI command `viam org list`. +You can find the model information on the [**MODELS** tab](https://app.viam.com/models). ```sh {class="command-line" data-prompt="$" data-output="2-18"} viam infer --binary-data-id --model-name --model-org-id --model-version "2025-04-14T16-38-25" --org-id @@ -151,5 +162,7 @@ Bounding Box Format: [x_min, y_min, x_max, y_max] No annotations. ``` -`infer` returns a list of detected classes or bounding boxes depending on the output of the ML model you specified, as well as a list of confidence values for those classes or boxes. -This method returns bounding box output using proportional coordinates between 0 and 1, with the origin `(0, 0)` in the top left of the image and `(1, 1)` in the bottom right. +The command returns a list of detected classes or bounding boxes depending on the output of the ML model you specified, as well as a list of confidence values for those classes or boxes. +The bounding box output uses proportional coordinates between 0 and 1, with the origin `(0, 0)` in the top left of the image and `(1, 1)` in the bottom right. + +For more information, see [`viam infer`](/dev/tools/cli/#infer). diff --git a/docs/data-ai/reference/triggers-configuration.md b/docs/data-ai/reference/triggers-configuration.md index e69f6b2c83..e087e8bcf2 100644 --- a/docs/data-ai/reference/triggers-configuration.md +++ b/docs/data-ai/reference/triggers-configuration.md @@ -47,24 +47,24 @@ The following template demonstrates the structure of a JSON configuration for a The following template demonstrates the structure of a JSON configuration for a trigger that alerts when data syncs to the Viam cloud: ```json {class="line-numbers linkable-line-numbers"} - "triggers": [ - { - "name": "", - "event": { - "type": "part_data_ingested", - "data_ingested": { - "data_types": ["binary", "tabular", "file", "unspecified"] - } - }, - "notifications": [ - { - "type": "", - "value": "", - "seconds_between_notifications": - } - ] - } - ] +"triggers": [ + { + "name": "", + "event": { + "type": "part_data_ingested", + "data_ingested": { + "data_types": ["binary", "tabular", "file", "unspecified"] + } + }, + "notifications": [ + { + "type": "", + "value": "", + "seconds_between_notifications": + } + ] + } +] ``` ### Conditional trigger template diff --git a/static/include/examples-generated/run-inference.snippet.run-inference.go b/static/include/examples-generated/run-inference.snippet.run-inference.go new file mode 100644 index 0000000000..9fe5c28070 --- /dev/null +++ b/static/include/examples-generated/run-inference.snippet.run-inference.go @@ -0,0 +1,177 @@ +package main + +import ( + "bytes" + "context" + "fmt" + "image" + "image/jpeg" + + "gorgonia.org/tensor" + + "go.viam.com/rdk/logging" + "go.viam.com/rdk/ml" + "go.viam.com/rdk/robot/client" + "go.viam.com/rdk/components/camera" + "go.viam.com/rdk/services/mlmodel" + "go.viam.com/rdk/utils" + "go.viam.com/utils/rpc" +) + +func main() { + apiKey := "" + apiKeyID := "" + machineAddress := "" + mlModelName := "" + cameraName := "" + + + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + machine, err := client.New( + context.Background(), + machineAddress, + logger, + client.WithDialOptions(rpc.WithEntityCredentials( + apiKeyID, + rpc.Credentials{ + Type: rpc.CredentialsTypeAPIKey, + Payload: apiKey, + })), + ) + if err != nil { + logger.Fatal(err) + } + + // Capture image from camera + cam, err := camera.FromRobot(machine, cameraName) + if err != nil { + logger.Fatal(err) + } + + imageData, _, err := cam.Image(ctx, utils.MimeTypeJPEG, nil) + if err != nil { + logger.Fatal(err) + } + + // Decode the image data to get the actual image + img, err := jpeg.Decode(bytes.NewReader(imageData)) + if err != nil { + logger.Fatal(err) + } + + // Get ML model metadata to understand input requirements + mlModel, err := mlmodel.FromRobot(machine, mlModelName) + if err != nil { + logger.Fatal(err) + } + + metadata, err := mlModel.Metadata(ctx) + if err != nil { + logger.Fatal(err) + } + + // Get expected input shape and type from metadata + var expectedShape []int + var expectedDtype tensor.Dtype + var expectedName string + + if len(metadata.Inputs) > 0 { + inputInfo := metadata.Inputs[0] + expectedShape = inputInfo.Shape + expectedName = inputInfo.Name + + // Convert data type string to tensor.Dtype + switch inputInfo.DataType { + case "uint8": + expectedDtype = tensor.Uint8 + case "float32": + expectedDtype = tensor.Float32 + default: + expectedDtype = tensor.Float32 // Default to float32 + } + } else { + logger.Fatal("No input info found in model metadata") + } + + // Resize image to expected dimensions + bounds := img.Bounds() + width := bounds.Dx() + height := bounds.Dy() + + // Extract expected dimensions + if len(expectedShape) != 4 || expectedShape[0] != 1 || expectedShape[3] != 3 { + logger.Fatal("Unexpected input shape format") + } + + expectedHeight := expectedShape[1] + expectedWidth := expectedShape[2] + + // Create a new image with the expected dimensions + resizedImg := image.NewRGBA(image.Rect(0, 0, expectedWidth, expectedHeight)) + + // Simple nearest neighbor resize + for y := 0; y < expectedHeight; y++ { + for x := 0; x < expectedWidth; x++ { + srcX := x * width / expectedWidth + srcY := y * height / expectedHeight + resizedImg.Set(x, y, img.At(srcX, srcY)) + } + } + + // Convert image to tensor data + tensorData := make([]float32, 1*expectedHeight*expectedWidth*3) + idx := 0 + + for y := 0; y < expectedHeight; y++ { + for x := 0; x < expectedWidth; x++ { + r, g, b, _ := resizedImg.At(x, y).RGBA() + + // Convert from 16-bit to 8-bit and normalize to [0, 1] for float32 + if expectedDtype == tensor.Float32 { + tensorData[idx] = float32(r>>8) / 255.0 // R + tensorData[idx+1] = float32(g>>8) / 255.0 // G + tensorData[idx+2] = float32(b>>8) / 255.0 // B + } else { + // For uint8, we need to create a uint8 slice + logger.Fatal("uint8 tensor creation not implemented in this example") + } + idx += 3 + } + } + + // Create input tensor + var inputTensor tensor.Tensor + if expectedDtype == tensor.Float32 { + inputTensor = tensor.New( + tensor.WithShape(1, expectedHeight, expectedWidth, 3), + tensor.WithBacking(tensorData), + tensor.Of(tensor.Float32), + ) + } else { + logger.Fatal("Only float32 tensors are supported in this example") + } + + // Convert tensor.Tensor to *tensor.Dense for ml.Tensors + denseTensor, ok := inputTensor.(*tensor.Dense) + if !ok { + logger.Fatal("Failed to convert inputTensor to *tensor.Dense") + } + + inputTensors := ml.Tensors{ + expectedName: denseTensor, + } + + outputTensors, err := mlModel.Infer(ctx, inputTensors) + if err != nil { + logger.Fatal(err) + } + + fmt.Printf("Output tensors: %v\n", outputTensors) + + err = machine.Close(ctx) + if err != nil { + logger.Fatal(err) + } +} diff --git a/static/include/examples-generated/run-inference.snippet.run-inference.py b/static/include/examples-generated/run-inference.snippet.run-inference.py new file mode 100644 index 0000000000..bccf0a8b60 --- /dev/null +++ b/static/include/examples-generated/run-inference.snippet.run-inference.py @@ -0,0 +1,91 @@ +import asyncio +import numpy as np + +from PIL import Image +from viam.components.camera import Camera +from viam.media.utils.pil import viam_to_pil_image +from viam.robot.client import RobotClient +from viam.services.mlmodel import MLModelClient + +# Configuration constants – replace with your actual values +API_KEY = "" # API key, find or create in your organization settings +API_KEY_ID = "" # API key ID, find or create in your organization settings +MACHINE_ADDRESS = "" # the address of the machine you want to capture images from +ML_MODEL_NAME = "" # the name of the ML model you want to use +CAMERA_NAME = "" # the name of the camera you want to capture images from + + +async def connect_machine() -> RobotClient: + """Establish a connection to the robot using the robot address.""" + machine_opts = RobotClient.Options.with_api_key( + api_key=API_KEY, + api_key_id=API_KEY_ID + ) + return await RobotClient.at_address(MACHINE_ADDRESS, machine_opts) + +async def main() -> int: + machine = await connect_machine() + + camera = Camera.from_robot(machine, CAMERA_NAME) + ml_model = MLModelClient.from_robot(machine, ML_MODEL_NAME) + + # Get ML model metadata to understand input requirements + metadata = await ml_model.metadata() + + # Capture image + image_frame = await camera.get_image() + + # Convert ViamImage to PIL Image first + pil_image = viam_to_pil_image(image_frame) + # Convert PIL Image to numpy array + image_array = np.array(pil_image) + + # Get expected input shape from metadata + expected_shape = list(metadata.input_info[0].shape) + expected_dtype = metadata.input_info[0].data_type + expected_name = metadata.input_info[0].name + + if not expected_shape: + print("No input info found for 'image'") + return 1 + + if len(expected_shape) == 4 and expected_shape[0] == 1 and expected_shape[3] == 3: + expected_height = expected_shape[1] + expected_width = expected_shape[2] + + # Resize to expected dimensions + if image_array.shape[:2] != (expected_height, expected_width): + pil_image_resized = pil_image.resize((expected_width, expected_height)) + image_array = np.array(pil_image_resized) + else: + print(f"Unexpected input shape format.") + return 1 + + # Add batch dimension and ensure correct shape + image_data = np.expand_dims(image_array, axis=0) + + # Ensure the data type matches expected type + if expected_dtype == "uint8": + image_data = image_data.astype(np.uint8) + elif expected_dtype == "float32": + # Convert to float32 and normalize to [0, 1] range + image_data = image_data.astype(np.float32) / 255.0 + else: + # Default to float32 with normalization + image_data = image_data.astype(np.float32) / 255.0 + + # Create the input tensors dictionary + input_tensors = { + expected_name: image_data + } + + output_tensors = await ml_model.infer(input_tensors) + print(f"Output tensors:") + for key, value in output_tensors.items(): + print(f"{key}: shape={value.shape}, dtype={value.dtype}") + + await machine.close() + return 0 + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/static/include/examples-generated/run-vision-service.snippet.run-vision-service.go b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.go new file mode 100644 index 0000000000..58e321d716 --- /dev/null +++ b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.go @@ -0,0 +1,75 @@ +package main + +import ( + "context" + "fmt" + "image/jpeg" + "bytes" + + "go.viam.com/rdk/logging" + "go.viam.com/rdk/robot/client" + "go.viam.com/rdk/services/vision" + "go.viam.com/rdk/components/camera" + "go.viam.com/rdk/utils" + "go.viam.com/utils/rpc" +) + +func main() { + apiKey := "" + apiKeyID := "" + machineAddress := "" + classifierName := "" + cameraName := "" + + + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + machine, err := client.New( + context.Background(), + machineAddress, + logger, + client.WithDialOptions(rpc.WithEntityCredentials( + apiKeyID, + rpc.Credentials{ + Type: rpc.CredentialsTypeAPIKey, + Payload: apiKey, + })), + ) + if err != nil { + logger.Fatal(err) + } + + // Capture image from camera + cam, err := camera.FromRobot(machine, cameraName) + if err != nil { + logger.Fatal(err) + } + + imageData, _, err := cam.Image(ctx, utils.MimeTypeJPEG, nil) + if err != nil { + logger.Fatal(err) + } + + // Convert binary data to image.Image + img, err := jpeg.Decode(bytes.NewReader(imageData)) + if err != nil { + logger.Fatal(err) + } + + // Get classifications using the image + classifier, err := vision.FromRobot(machine, classifierName) + if err != nil { + logger.Fatal(err) + } + + classifications, err := classifier.Classifications(ctx, img, 2, nil) + if err != nil { + logger.Fatal(err) + } + + err = machine.Close(ctx) + if err != nil { + logger.Fatal(err) + } +} diff --git a/static/include/examples-generated/run-vision-service.snippet.run-vision-service.py b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.py new file mode 100644 index 0000000000..6d367a2d1a --- /dev/null +++ b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.py @@ -0,0 +1,43 @@ +import asyncio +import numpy as np + +from PIL import Image +from viam.components.camera import Camera +from viam.media.utils.pil import viam_to_pil_image +from viam.robot.client import RobotClient +from viam.services.vision import VisionClient + +# Configuration constants – replace with your actual values +API_KEY = "" # API key, find or create in your organization settings +API_KEY_ID = "" # API key ID, find or create in your organization settings +MACHINE_ADDRESS = "" # the address of the machine you want to capture images from +CLASSIFIER_NAME = "" # the name of the classifier you want to use +CAMERA_NAME = "" # the name of the camera you want to capture images from + + +async def connect_machine() -> RobotClient: + """Establish a connection to the robot using the robot address.""" + machine_opts = RobotClient.Options.with_api_key( + api_key=API_KEY, + api_key_id=API_KEY_ID + ) + return await RobotClient.at_address(MACHINE_ADDRESS, machine_opts) + +async def main() -> int: + machine = await connect_machine() + + camera = Camera.from_robot(machine, CAMERA_NAME) + classifier = VisionClient.from_robot(machine, CLASSIFIER_NAME) + + # Capture image + image_frame = await camera.get_image(mime_type="image/jpeg") + + # Get tags using the ViamImage (not the PIL image) + tags = await classifier.get_classifications( + image=image_frame, image_format="image/jpeg", count=2) + + await machine.close() + return 0 + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/static/include/examples-generated/run-vision-service.snippet.run-vision-service.ts b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.ts new file mode 100644 index 0000000000..98b7f98857 --- /dev/null +++ b/static/include/examples-generated/run-vision-service.snippet.run-vision-service.ts @@ -0,0 +1,46 @@ +import { createRobotClient, RobotClient, VisionClient, CameraClient } from "@viamrobotics/sdk"; + +// Configuration constants – replace with your actual values +let API_KEY = ""; // API key, find or create in your organization settings +let API_KEY_ID = ""; // API key ID, find or create in your organization settings +let MACHINE_ADDRESS = ""; // the address of the machine you want to capture images from +let CLASSIFIER_NAME = ""; // the name of the classifier you want to use +let CAMERA_NAME = ""; // the name of the camera you want to capture images from + +async function connectMachine(): Promise { + // Establish a connection to the robot using the machine address + return await createRobotClient({ + host: MACHINE_ADDRESS, + credentials: { + type: 'api-key', + payload: API_KEY, + authEntity: API_KEY_ID, + }, + signalingAddress: 'https://app.viam.com:443', + }); +} + +async function main(): Promise { + const machine = await connectMachine(); + const camera = new CameraClient(machine, CAMERA_NAME); + const classifier = new VisionClient(machine, CLASSIFIER_NAME); + + // Capture image + const imageFrame = await camera.getImage(); + + // Get tags using the image + const tags = await classifier.getClassifications( + imageFrame, + imageFrame.width ?? 0, + imageFrame.height ?? 0, + imageFrame.mimeType ?? "", + 2 + ); + + return 0; +} + +main().catch((error) => { + console.error("Script failed:", error); + process.exit(1); +}); diff --git a/static/include/examples/fleet-api/fleet-issue-1.go b/static/include/examples/fleet-api/fleet-issue-1.go new file mode 100644 index 0000000000..2cbd55efd2 --- /dev/null +++ b/static/include/examples/fleet-api/fleet-issue-1.go @@ -0,0 +1,88 @@ +package main + +import ( + "context" + "fmt" + "os" + + "go.viam.com/rdk/app" + "go.viam.com/rdk/logging" +) + +// Configuration constants – replace with your actual values +var ( + API_KEY = "" // API key, find or create in your organization settings + API_KEY_ID = "" // API key ID, find or create in your organization settings + ORG_ID = "" // Organization ID, find or create in your organization settings +) + +func main() { + ORG_ID = os.Getenv("TEST_ORG_ID") + API_KEY = os.Getenv("VIAM_API_KEY") + API_KEY_ID = os.Getenv("VIAM_API_KEY_ID") + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + // Make a ViamClient + viamClient, err := app.CreateViamClientWithAPIKey( + ctx, app.Options{}, API_KEY, API_KEY_ID, logger) + if err != nil { + logger.Fatal(err) + } + defer viamClient.Close() + + // Instantiate an AppClient called "cloud" + // to run fleet management API methods on + cloud := viamClient.AppClient() + + // ISSUE 1: CREATE & DELETE REGISTRY ITEM + + // Create registry item + err = cloud.CreateRegistryItem(ctx, ORG_ID, "new-registry-item-12", app.PackageTypeMLModel) + if err != nil { + logger.Fatal(err) + } + + // Get number of registry items + registryItems, err := cloud.ListRegistryItems( + ctx, + &ORG_ID, + []app.PackageType{app.PackageTypeMLModel}, + []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + []string{"linux/any"}, + []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + &app.ListRegistryItemsOptions{}) + if err != nil { + logger.Fatal(err) + } + numRegistryItems := len(registryItems) + fmt.Println("Number of registry items:", numRegistryItems) + if numRegistryItems <= 0 { + logger.Fatal("Expected > 0 registry items") + } + + // DOES NOT SEEM TO DELETE THE REGISTRY ITEM + // Delete registry item + err = cloud.DeleteRegistryItem(ctx, "docs-test:new-registry-item-12") + if err != nil { + logger.Fatal(err) + } + + // Get number of registry items + registryItems, err = cloud.ListRegistryItems( + ctx, + &ORG_ID, + []app.PackageType{app.PackageTypeMLModel}, + []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + []string{"linux/any"}, + []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + &app.ListRegistryItemsOptions{}) + if err != nil { + logger.Fatal(err) + } + fmt.Println("Number of registry items:", len(registryItems)) + if numRegistryItems - 1 != len(registryItems) { + logger.Fatal("Expected 1 fewer registry item after deletion") + } + +} diff --git a/static/include/examples/fleet-api/fleet-issue-2.go b/static/include/examples/fleet-api/fleet-issue-2.go new file mode 100644 index 0000000000..c12eb0558f --- /dev/null +++ b/static/include/examples/fleet-api/fleet-issue-2.go @@ -0,0 +1,76 @@ +package main + +import ( + "context" + "fmt" + "os" + + "go.viam.com/rdk/app" + "go.viam.com/rdk/logging" +) + +// Configuration constants – replace with your actual values +var ( + API_KEY = "" // API key, find or create in your organization settings + API_KEY_ID = "" // API key ID, find or create in your organization settings + ORG_ID = "" // Organization ID, find or create in your organization settings + LOCATION_ID = "" // Location ID, find or create in your organization settings +) + +func main() { + // :remove-start: + ORG_ID = os.Getenv("TEST_ORG_ID") + API_KEY = os.Getenv("VIAM_API_KEY") + API_KEY_ID = os.Getenv("VIAM_API_KEY_ID") + LOCATION_ID = "pg5q3j3h95" + // :remove-end: + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + // Make a ViamClient + viamClient, err := app.CreateViamClientWithAPIKey( + ctx, app.Options{}, API_KEY, API_KEY_ID, logger) + if err != nil { + logger.Fatal(err) + } + defer viamClient.Close() + + // Instantiate an AppClient called "cloud" + // to run fleet management API methods on + cloud := viamClient.AppClient() + + // ISSUE 2: Add role + // User ID: d3bcb264-1a60-406b-8a79-9e43da4f3c9d + // 2025-09-02T10:31:21.526Z ERROR client fleet-api/fleet-issues.go:113 rpc error: code = InvalidArgument desc = requestID=9806ca33a0007d462e545f16a7cc0ec0: provided invalid authorization id 'location_owner' for resource type 'location' and authorization type 'owner' + + memberList, _, err := cloud.ListOrganizationMembers(ctx, ORG_ID) + if err != nil { + logger.Fatal(err) + } + + // Add role + userID := memberList[1].UserID + fmt.Println("User ID:", userID) + + err = cloud.AddRole( + ctx, + ORG_ID, + userID, + app.AuthRoleOwner, + app.AuthResourceTypeLocation, + LOCATION_ID, + ) + if err != nil { + logger.Fatal(err) + } + + // Change role + fmt.Printf(userID) + err = cloud.ChangeRole(ctx, &app.Authorization{}, ORG_ID, userID, app.AuthRoleOwner, app.AuthResourceTypeOrganization, ORG_ID) + if err != nil { + logger.Fatal(err) + } + + // d3bcb264-1a60-406b-8a79-9e43da4f3c9d2025-09-02T10:33:43.956Z ERROR client fleet-api/fleet-issues.go:122 rpc error: code = InvalidArgument desc = requestID=df8a74809aad4dfd156a7c2bad698d23: missing required 'identity_id' + +} diff --git a/static/include/examples/fleet-api/fleet-issue-3.go b/static/include/examples/fleet-api/fleet-issue-3.go new file mode 100644 index 0000000000..1bda39812b --- /dev/null +++ b/static/include/examples/fleet-api/fleet-issue-3.go @@ -0,0 +1,168 @@ +package main + +import ( + "context" + "fmt" + "os" + + "go.viam.com/rdk/app" + "go.viam.com/rdk/logging" +) + +// Configuration constants – replace with your actual values +var ( + API_KEY = "" // API key, find or create in your organization settings + API_KEY_ID = "" // API key ID, find or create in your organization settings + ORG_ID = "" // Organization ID, find or create in your organization settings + EMAIL_ADDRESS = "" // Email address of the user to get the user id for + LOCATION_ID = "" // Location ID, find or create in your organization settings +) + +func main() { + // :remove-start: + ORG_ID = os.Getenv("TEST_ORG_ID") + API_KEY = os.Getenv("VIAM_API_KEY") + API_KEY_ID = os.Getenv("VIAM_API_KEY_ID") + LOCATION_ID = "pg5q3j3h95" + // :remove-end: + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + // Make a ViamClient + viamClient, err := app.CreateViamClientWithAPIKey( + ctx, app.Options{}, API_KEY, API_KEY_ID, logger) + if err != nil { + logger.Fatal(err) + } + defer viamClient.Close() + + // Instantiate an AppClient called "cloud" + // to run fleet management API methods on + cloud := viamClient.AppClient() + + // ISSUE 1: CREATE & DELETE REGISTRY ITEM + + // // Create registry item + // err = cloud.CreateRegistryItem(ctx, ORG_ID, "new-registry-item-12", app.PackageTypeMLModel) + // if err != nil { + // logger.Fatal(err) + // } + + // // Get number of registry items + // registryItems, err := cloud.ListRegistryItems( + // ctx, + // &ORG_ID, + // []app.PackageType{app.PackageTypeMLModel}, + // []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + // []string{"linux/any"}, + // []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + // &app.ListRegistryItemsOptions{}) + // if err != nil { + // logger.Fatal(err) + // } + // numRegistryItems := len(registryItems) + // fmt.Println("Number of registry items:", numRegistryItems) + // if numRegistryItems <= 0 { + // logger.Fatal("Expected > 0 registry items") + // } + + // // DOES NOT SEEM TO DELETE THE REGISTRY ITEM + // // Delete registry item + // err = cloud.DeleteRegistryItem(ctx, "docs-test:new-registry-item-12") + // if err != nil { + // logger.Fatal(err) + // } + + // // Get number of registry items + // registryItems, err = cloud.ListRegistryItems( + // ctx, + // &ORG_ID, + // []app.PackageType{app.PackageTypeMLModel}, + // []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + // []string{"linux/any"}, + // []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + // &app.ListRegistryItemsOptions{}) + // if err != nil { + // logger.Fatal(err) + // } + // fmt.Println("Number of registry items:", len(registryItems)) + // if numRegistryItems - 1 != len(registryItems) { + // logger.Fatal("Expected 1 fewer registry item after deletion") + // } + + // ISSUE 2: Add role + // User ID: d3bcb264-1a60-406b-8a79-9e43da4f3c9d + // 2025-09-02T10:31:21.526Z ERROR client fleet-api/fleet-issues.go:113 rpc error: code = InvalidArgument desc = requestID=9806ca33a0007d462e545f16a7cc0ec0: provided invalid authorization id 'location_owner' for resource type 'location' and authorization type 'owner' + + memberList, _, err := cloud.ListOrganizationMembers(ctx, ORG_ID) + if err != nil { + logger.Fatal(err) + } + + // Add role + userID := memberList[1].UserID + fmt.Println("User ID:", userID) + + // err = cloud.AddRole( + // ctx, + // ORG_ID, + // userID, + // app.AuthRoleOwner, + // app.AuthResourceTypeLocation, + // LOCATION_ID, + // ) + // if err != nil { + // logger.Fatal(err) + // } + + // Change role + fmt.Printf(userID) + // err = cloud.ChangeRole(ctx, &app.Authorization{}, ORG_ID, userID, app.AuthRoleOwner, app.AuthResourceTypeOrganization, ORG_ID) + // if err != nil { + // logger.Fatal(err) + // } + + // d3bcb264-1a60-406b-8a79-9e43da4f3c9d2025-09-02T10:33:43.956Z ERROR client fleet-api/fleet-issues.go:122 rpc error: code = InvalidArgument desc = requestID=df8a74809aad4dfd156a7c2bad698d23: missing required 'identity_id' + + // ISSUE 3: It seems we can't create keys because we can't create APIKeyAuthorization structs + + // Create API key + // Since APIKeyAuthorization has unexported fields, we can't construct it directly + // apiKey, apiKeyID, err := cloud.CreateKey(ctx, ORG_ID, []app.APIKeyAuthorization{{ + // Role: app.AuthRoleOwner, + // ResourceType: app.AuthResourceTypeLocation, + // ResourceID: LOCATION_ID, + // }}, "mytestkey") + // if err != nil { + // logger.Fatal(err) + // } + // if apiKey == "" { + // logger.Fatal("API key should not be empty") + // } + // if apiKeyID == "" { + // logger.Fatal("API key ID should not be empty") + // } + + // ISSUE 4: RenameKey does not return the key ID + apiKeyID2, apiKey2, err := cloud.CreateKeyFromExistingKeyAuthorizations(ctx, newAPIKeyID) + if err != nil { + logger.Fatal(err) + } + if apiKey2 == "" { + logger.Fatal("API key 2 should not be empty") + } + if apiKeyID2 == "" { + logger.Fatal("API key ID 2 should not be empty") + } + fmt.Printf("API key id 2: %+v\n", apiKeyID2) + + apiKeyID3, apiKeyName3, err := cloud.RenameKey(ctx, apiKeyID2, "mytestkey2newName") + if err != nil { + logger.Fatal(err) + } + if apiKeyName3 != "mytestkey2newName" { + logger.Fatal("API key 3 should be renamed") + } + fmt.Printf("API key id 3: %+v\n", apiKeyID3) + +} diff --git a/static/include/examples/fleet-api/fleet-issue-4.go b/static/include/examples/fleet-api/fleet-issue-4.go new file mode 100644 index 0000000000..1bda39812b --- /dev/null +++ b/static/include/examples/fleet-api/fleet-issue-4.go @@ -0,0 +1,168 @@ +package main + +import ( + "context" + "fmt" + "os" + + "go.viam.com/rdk/app" + "go.viam.com/rdk/logging" +) + +// Configuration constants – replace with your actual values +var ( + API_KEY = "" // API key, find or create in your organization settings + API_KEY_ID = "" // API key ID, find or create in your organization settings + ORG_ID = "" // Organization ID, find or create in your organization settings + EMAIL_ADDRESS = "" // Email address of the user to get the user id for + LOCATION_ID = "" // Location ID, find or create in your organization settings +) + +func main() { + // :remove-start: + ORG_ID = os.Getenv("TEST_ORG_ID") + API_KEY = os.Getenv("VIAM_API_KEY") + API_KEY_ID = os.Getenv("VIAM_API_KEY_ID") + LOCATION_ID = "pg5q3j3h95" + // :remove-end: + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + // Make a ViamClient + viamClient, err := app.CreateViamClientWithAPIKey( + ctx, app.Options{}, API_KEY, API_KEY_ID, logger) + if err != nil { + logger.Fatal(err) + } + defer viamClient.Close() + + // Instantiate an AppClient called "cloud" + // to run fleet management API methods on + cloud := viamClient.AppClient() + + // ISSUE 1: CREATE & DELETE REGISTRY ITEM + + // // Create registry item + // err = cloud.CreateRegistryItem(ctx, ORG_ID, "new-registry-item-12", app.PackageTypeMLModel) + // if err != nil { + // logger.Fatal(err) + // } + + // // Get number of registry items + // registryItems, err := cloud.ListRegistryItems( + // ctx, + // &ORG_ID, + // []app.PackageType{app.PackageTypeMLModel}, + // []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + // []string{"linux/any"}, + // []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + // &app.ListRegistryItemsOptions{}) + // if err != nil { + // logger.Fatal(err) + // } + // numRegistryItems := len(registryItems) + // fmt.Println("Number of registry items:", numRegistryItems) + // if numRegistryItems <= 0 { + // logger.Fatal("Expected > 0 registry items") + // } + + // // DOES NOT SEEM TO DELETE THE REGISTRY ITEM + // // Delete registry item + // err = cloud.DeleteRegistryItem(ctx, "docs-test:new-registry-item-12") + // if err != nil { + // logger.Fatal(err) + // } + + // // Get number of registry items + // registryItems, err = cloud.ListRegistryItems( + // ctx, + // &ORG_ID, + // []app.PackageType{app.PackageTypeMLModel}, + // []app.Visibility{app.VisibilityPrivate, app.VisibilityPublic}, + // []string{"linux/any"}, + // []app.RegistryItemStatus{app.RegistryItemStatusPublished}, + // &app.ListRegistryItemsOptions{}) + // if err != nil { + // logger.Fatal(err) + // } + // fmt.Println("Number of registry items:", len(registryItems)) + // if numRegistryItems - 1 != len(registryItems) { + // logger.Fatal("Expected 1 fewer registry item after deletion") + // } + + // ISSUE 2: Add role + // User ID: d3bcb264-1a60-406b-8a79-9e43da4f3c9d + // 2025-09-02T10:31:21.526Z ERROR client fleet-api/fleet-issues.go:113 rpc error: code = InvalidArgument desc = requestID=9806ca33a0007d462e545f16a7cc0ec0: provided invalid authorization id 'location_owner' for resource type 'location' and authorization type 'owner' + + memberList, _, err := cloud.ListOrganizationMembers(ctx, ORG_ID) + if err != nil { + logger.Fatal(err) + } + + // Add role + userID := memberList[1].UserID + fmt.Println("User ID:", userID) + + // err = cloud.AddRole( + // ctx, + // ORG_ID, + // userID, + // app.AuthRoleOwner, + // app.AuthResourceTypeLocation, + // LOCATION_ID, + // ) + // if err != nil { + // logger.Fatal(err) + // } + + // Change role + fmt.Printf(userID) + // err = cloud.ChangeRole(ctx, &app.Authorization{}, ORG_ID, userID, app.AuthRoleOwner, app.AuthResourceTypeOrganization, ORG_ID) + // if err != nil { + // logger.Fatal(err) + // } + + // d3bcb264-1a60-406b-8a79-9e43da4f3c9d2025-09-02T10:33:43.956Z ERROR client fleet-api/fleet-issues.go:122 rpc error: code = InvalidArgument desc = requestID=df8a74809aad4dfd156a7c2bad698d23: missing required 'identity_id' + + // ISSUE 3: It seems we can't create keys because we can't create APIKeyAuthorization structs + + // Create API key + // Since APIKeyAuthorization has unexported fields, we can't construct it directly + // apiKey, apiKeyID, err := cloud.CreateKey(ctx, ORG_ID, []app.APIKeyAuthorization{{ + // Role: app.AuthRoleOwner, + // ResourceType: app.AuthResourceTypeLocation, + // ResourceID: LOCATION_ID, + // }}, "mytestkey") + // if err != nil { + // logger.Fatal(err) + // } + // if apiKey == "" { + // logger.Fatal("API key should not be empty") + // } + // if apiKeyID == "" { + // logger.Fatal("API key ID should not be empty") + // } + + // ISSUE 4: RenameKey does not return the key ID + apiKeyID2, apiKey2, err := cloud.CreateKeyFromExistingKeyAuthorizations(ctx, newAPIKeyID) + if err != nil { + logger.Fatal(err) + } + if apiKey2 == "" { + logger.Fatal("API key 2 should not be empty") + } + if apiKeyID2 == "" { + logger.Fatal("API key ID 2 should not be empty") + } + fmt.Printf("API key id 2: %+v\n", apiKeyID2) + + apiKeyID3, apiKeyName3, err := cloud.RenameKey(ctx, apiKeyID2, "mytestkey2newName") + if err != nil { + logger.Fatal(err) + } + if apiKeyName3 != "mytestkey2newName" { + logger.Fatal("API key 3 should be renamed") + } + fmt.Printf("API key id 3: %+v\n", apiKeyID3) + +} diff --git a/static/include/examples/go.mod b/static/include/examples/go.mod index 21ab063a30..400d67071f 100644 --- a/static/include/examples/go.mod +++ b/static/include/examples/go.mod @@ -6,6 +6,7 @@ require ( github.com/golang/geo v0.0.0-20230421003525-6adc56603217 go.viam.com/rdk v0.91.0 go.viam.com/utils v0.1.164 + gorgonia.org/tensor v0.9.24 ) require ( @@ -21,6 +22,7 @@ require ( git.sr.ht/~sbinet/gg v0.6.0 // indirect github.com/a8m/envsubst v1.4.2 // indirect github.com/ajstarks/svgo v0.0.0-20211024235047-1546f124cd8b // indirect + github.com/apache/arrow/go/arrow v0.0.0-20201229220542-30ce2eb5d4dc // indirect github.com/aybabtme/uniplot v0.0.0-20151203143629-039c559e5e7e // indirect github.com/benbjohnson/clock v1.3.5 // indirect github.com/bep/debounce v1.2.1 // indirect @@ -31,6 +33,8 @@ require ( github.com/cenkalti/backoff v2.2.1+incompatible // indirect github.com/cenkalti/backoff/v4 v4.3.0 // indirect github.com/chenzhekl/goply v0.0.0-20190930133256-258c2381defd // indirect + github.com/chewxy/hm v1.0.0 // indirect + github.com/chewxy/math32 v1.0.8 // indirect github.com/cloudwego/base64x v0.1.5 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect @@ -52,12 +56,14 @@ require ( github.com/go-logr/stdr v1.2.2 // indirect github.com/go-viper/mapstructure/v2 v2.4.0 // indirect github.com/goccy/go-json v0.10.2 // indirect + github.com/gogo/protobuf v1.3.2 // indirect github.com/golang-jwt/jwt/v4 v4.5.2 // indirect github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/protobuf v1.5.4 // indirect github.com/golang/snappy v0.0.4 // indirect github.com/gonuts/binary v0.2.0 // indirect + github.com/google/flatbuffers v2.0.6+incompatible // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/s2a-go v0.1.8 // indirect github.com/google/uuid v1.6.0 // indirect @@ -125,6 +131,7 @@ require ( github.com/xdg-go/scram v1.1.2 // indirect github.com/xdg-go/stringprep v1.0.4 // indirect github.com/xfmoulet/qoi v0.2.0 // indirect + github.com/xtgo/set v1.0.0 // indirect github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect github.com/zhuyie/golzf v0.0.0-20161112031142-8387b0307ade // indirect github.com/zitadel/oidc/v3 v3.37.0 // indirect @@ -144,6 +151,7 @@ require ( go.uber.org/zap v1.27.0 // indirect go.viam.com/api v0.1.475 // indirect go.viam.com/test v1.2.4 // indirect + go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect golang.org/x/arch v0.19.0 // indirect golang.org/x/crypto v0.41.0 // indirect golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c // indirect @@ -156,6 +164,7 @@ require ( golang.org/x/text v0.28.0 // indirect golang.org/x/time v0.6.0 // indirect golang.org/x/tools v0.35.0 // indirect + golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect gonum.org/v1/gonum v0.16.0 // indirect gonum.org/v1/plot v0.15.2 // indirect google.golang.org/api v0.196.0 // indirect @@ -166,5 +175,7 @@ require ( google.golang.org/protobuf v1.36.8 // indirect gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect + gorgonia.org/vecf32 v0.9.0 // indirect + gorgonia.org/vecf64 v0.9.0 // indirect nhooyr.io/websocket v1.8.7 // indirect ) diff --git a/static/include/examples/go.sum b/static/include/examples/go.sum index 185e91cedd..2567260a78 100644 --- a/static/include/examples/go.sum +++ b/static/include/examples/go.sum @@ -138,6 +138,7 @@ github.com/chenzhekl/goply v0.0.0-20190930133256-258c2381defd h1:S0onsSZ3RawTrm4 github.com/chenzhekl/goply v0.0.0-20190930133256-258c2381defd/go.mod h1:P2dOeu3SNXtjA5VOH7tF0AnGm/eYrst9YA89b36c35I= github.com/chewxy/hm v1.0.0 h1:zy/TSv3LV2nD3dwUEQL2VhXeoXbb9QkpmdRAVUFiA6k= github.com/chewxy/hm v1.0.0/go.mod h1:qg9YI4q6Fkj/whwHR1D+bOGeF7SniIP40VweVepLjg0= +github.com/chewxy/math32 v1.0.0/go.mod h1:Miac6hA1ohdDUTagnvJy/q+aNnEk16qWUdb8ZVhvCN0= github.com/chewxy/math32 v1.0.8 h1:fU5E4Ec4Z+5RtRAi3TovSxUjQPkgRh+HbP7tKB2OFbM= github.com/chewxy/math32 v1.0.8/go.mod h1:dOB2rcuFrCn6UHrze36WSLVPKtzPMRAQvBvUwkSsLqs= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= @@ -365,6 +366,7 @@ github.com/gonuts/binary v0.2.0 h1:caITwMWAoQWlL0RNvv2lTU/AHqAJlVuu6nZmNgfbKW4= github.com/gonuts/binary v0.2.0/go.mod h1:kM+CtBrCGDSKdv8WXTuCUsw+loiy8f/QEI8YCCC0M/E= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/flatbuffers v1.11.0/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= github.com/google/flatbuffers v2.0.6+incompatible h1:XHFReMv7nFFusa+CEokzWbzaYocKXI6C7hdU5Kgh9Lw= github.com/google/flatbuffers v2.0.6+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= @@ -844,6 +846,7 @@ github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSS github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= github.com/stretchr/testify v1.1.4/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.2.0/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= @@ -1090,6 +1093,7 @@ golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/ golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= @@ -1168,6 +1172,7 @@ golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200909081042-eff7692f9009/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201024232916-9f70ab9862d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1354,6 +1359,7 @@ google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfG google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/genproto v0.0.0-20200911024640-645f7a48b24f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210126160654-44e461bb6506/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 h1:BulPr26Jqjnd4eYDVe+YvyR7Yc2vJGkO5/0UxD0/jZU= google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1/go.mod h1:hL97c3SYopEHblzpxRL4lSs523++l8DYxGM1FQiYmb4= @@ -1383,6 +1389,7 @@ google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAG google.golang.org/grpc v1.44.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= google.golang.org/grpc v1.75.0 h1:+TW+dqTd2Biwe6KKfhE5JpiYIBWq865PhKGSXiivqt4= google.golang.org/grpc v1.75.0/go.mod h1:JtPAzKiq4v1xcAB2hydNlWI2RnF85XXcV0mhKXr2ecQ= +google.golang.org/grpc/cmd/protoc-gen-go-grpc v0.0.0-20200910201057-6591123024b3/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= diff --git a/static/include/examples/run-inference/run-inference.go b/static/include/examples/run-inference/run-inference.go new file mode 100644 index 0000000000..22d65f794c --- /dev/null +++ b/static/include/examples/run-inference/run-inference.go @@ -0,0 +1,189 @@ +// :snippet-start: run-inference +package main + +import ( + "bytes" + "context" + "fmt" + "image" + "image/jpeg" + + "gorgonia.org/tensor" + // :remove-start: + "os" + // :remove-end: + + "go.viam.com/rdk/logging" + "go.viam.com/rdk/ml" + "go.viam.com/rdk/robot/client" + "go.viam.com/rdk/components/camera" + "go.viam.com/rdk/services/mlmodel" + "go.viam.com/rdk/utils" + "go.viam.com/utils/rpc" +) + +func main() { + apiKey := "" + apiKeyID := "" + machineAddress := "" + mlModelName := "" + cameraName := "" + + // :remove-start: + apiKey = os.Getenv("VIAM_API_KEY") + apiKeyID = os.Getenv("VIAM_API_KEY_ID") + machineAddress = "auto-machine-main.pg5q3j3h95.viam.cloud" + cameraName = "camera-1" + mlModelName = "mlmodel-1" + // :remove-end: + + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + machine, err := client.New( + context.Background(), + machineAddress, + logger, + client.WithDialOptions(rpc.WithEntityCredentials( + apiKeyID, + rpc.Credentials{ + Type: rpc.CredentialsTypeAPIKey, + Payload: apiKey, + })), + ) + if err != nil { + logger.Fatal(err) + } + + // Capture image from camera + cam, err := camera.FromRobot(machine, cameraName) + if err != nil { + logger.Fatal(err) + } + + imageData, _, err := cam.Image(ctx, utils.MimeTypeJPEG, nil) + if err != nil { + logger.Fatal(err) + } + + // Decode the image data to get the actual image + img, err := jpeg.Decode(bytes.NewReader(imageData)) + if err != nil { + logger.Fatal(err) + } + + // Get ML model metadata to understand input requirements + mlModel, err := mlmodel.FromRobot(machine, mlModelName) + if err != nil { + logger.Fatal(err) + } + + metadata, err := mlModel.Metadata(ctx) + if err != nil { + logger.Fatal(err) + } + + // Get expected input shape and type from metadata + var expectedShape []int + var expectedDtype tensor.Dtype + var expectedName string + + if len(metadata.Inputs) > 0 { + inputInfo := metadata.Inputs[0] + expectedShape = inputInfo.Shape + expectedName = inputInfo.Name + + // Convert data type string to tensor.Dtype + switch inputInfo.DataType { + case "uint8": + expectedDtype = tensor.Uint8 + case "float32": + expectedDtype = tensor.Float32 + default: + expectedDtype = tensor.Float32 // Default to float32 + } + } else { + logger.Fatal("No input info found in model metadata") + } + + // Resize image to expected dimensions + bounds := img.Bounds() + width := bounds.Dx() + height := bounds.Dy() + + // Extract expected dimensions + if len(expectedShape) != 4 || expectedShape[0] != 1 || expectedShape[3] != 3 { + logger.Fatal("Unexpected input shape format") + } + + expectedHeight := expectedShape[1] + expectedWidth := expectedShape[2] + + // Create a new image with the expected dimensions + resizedImg := image.NewRGBA(image.Rect(0, 0, expectedWidth, expectedHeight)) + + // Simple nearest neighbor resize + for y := 0; y < expectedHeight; y++ { + for x := 0; x < expectedWidth; x++ { + srcX := x * width / expectedWidth + srcY := y * height / expectedHeight + resizedImg.Set(x, y, img.At(srcX, srcY)) + } + } + + // Convert image to tensor data + tensorData := make([]float32, 1*expectedHeight*expectedWidth*3) + idx := 0 + + for y := 0; y < expectedHeight; y++ { + for x := 0; x < expectedWidth; x++ { + r, g, b, _ := resizedImg.At(x, y).RGBA() + + // Convert from 16-bit to 8-bit and normalize to [0, 1] for float32 + if expectedDtype == tensor.Float32 { + tensorData[idx] = float32(r>>8) / 255.0 // R + tensorData[idx+1] = float32(g>>8) / 255.0 // G + tensorData[idx+2] = float32(b>>8) / 255.0 // B + } else { + // For uint8, we need to create a uint8 slice + logger.Fatal("uint8 tensor creation not implemented in this example") + } + idx += 3 + } + } + + // Create input tensor + var inputTensor tensor.Tensor + if expectedDtype == tensor.Float32 { + inputTensor = tensor.New( + tensor.WithShape(1, expectedHeight, expectedWidth, 3), + tensor.WithBacking(tensorData), + tensor.Of(tensor.Float32), + ) + } else { + logger.Fatal("Only float32 tensors are supported in this example") + } + + // Convert tensor.Tensor to *tensor.Dense for ml.Tensors + denseTensor, ok := inputTensor.(*tensor.Dense) + if !ok { + logger.Fatal("Failed to convert inputTensor to *tensor.Dense") + } + + inputTensors := ml.Tensors{ + expectedName: denseTensor, + } + + outputTensors, err := mlModel.Infer(ctx, inputTensors) + if err != nil { + logger.Fatal(err) + } + + fmt.Printf("Output tensors: %v\n", outputTensors) + + err = machine.Close(ctx) + if err != nil { + logger.Fatal(err) + } +} +// :snippet-end: \ No newline at end of file diff --git a/static/include/examples/run-inference/run-inference.py b/static/include/examples/run-inference/run-inference.py new file mode 100644 index 0000000000..b45202b6f1 --- /dev/null +++ b/static/include/examples/run-inference/run-inference.py @@ -0,0 +1,103 @@ +# :snippet-start: run-inference +import asyncio +# :remove-start: +import os +# :remove-end: +import numpy as np + +from PIL import Image +from viam.components.camera import Camera +from viam.media.utils.pil import viam_to_pil_image +from viam.robot.client import RobotClient +from viam.services.mlmodel import MLModelClient + +# Configuration constants – replace with your actual values +API_KEY = "" # API key, find or create in your organization settings +API_KEY_ID = "" # API key ID, find or create in your organization settings +MACHINE_ADDRESS = "" # the address of the machine you want to capture images from +ML_MODEL_NAME = "" # the name of the ML model you want to use +CAMERA_NAME = "" # the name of the camera you want to capture images from + +# :remove-start: +API_KEY = os.environ["VIAM_API_KEY"] +API_KEY_ID = os.environ["VIAM_API_KEY_ID"] +MACHINE_ADDRESS = "auto-machine-main.pg5q3j3h95.viam.cloud" +CAMERA_NAME = "camera-1" +ML_MODEL_NAME = "mlmodel-1" +# :remove-end: + +async def connect_machine() -> RobotClient: + """Establish a connection to the robot using the robot address.""" + machine_opts = RobotClient.Options.with_api_key( + api_key=API_KEY, + api_key_id=API_KEY_ID + ) + return await RobotClient.at_address(MACHINE_ADDRESS, machine_opts) + +async def main() -> int: + machine = await connect_machine() + + camera = Camera.from_robot(machine, CAMERA_NAME) + ml_model = MLModelClient.from_robot(machine, ML_MODEL_NAME) + + # Get ML model metadata to understand input requirements + metadata = await ml_model.metadata() + + # Capture image + image_frame = await camera.get_image() + + # Convert ViamImage to PIL Image first + pil_image = viam_to_pil_image(image_frame) + # Convert PIL Image to numpy array + image_array = np.array(pil_image) + + # Get expected input shape from metadata + expected_shape = list(metadata.input_info[0].shape) + expected_dtype = metadata.input_info[0].data_type + expected_name = metadata.input_info[0].name + + if not expected_shape: + print("No input info found for 'image'") + return 1 + + if len(expected_shape) == 4 and expected_shape[0] == 1 and expected_shape[3] == 3: + expected_height = expected_shape[1] + expected_width = expected_shape[2] + + # Resize to expected dimensions + if image_array.shape[:2] != (expected_height, expected_width): + pil_image_resized = pil_image.resize((expected_width, expected_height)) + image_array = np.array(pil_image_resized) + else: + print(f"Unexpected input shape format.") + return 1 + + # Add batch dimension and ensure correct shape + image_data = np.expand_dims(image_array, axis=0) + + # Ensure the data type matches expected type + if expected_dtype == "uint8": + image_data = image_data.astype(np.uint8) + elif expected_dtype == "float32": + # Convert to float32 and normalize to [0, 1] range + image_data = image_data.astype(np.float32) / 255.0 + else: + # Default to float32 with normalization + image_data = image_data.astype(np.float32) / 255.0 + + # Create the input tensors dictionary + input_tensors = { + expected_name: image_data + } + + output_tensors = await ml_model.infer(input_tensors) + print(f"Output tensors:") + for key, value in output_tensors.items(): + print(f"{key}: shape={value.shape}, dtype={value.dtype}") + + await machine.close() + return 0 + +if __name__ == "__main__": + asyncio.run(main()) +# :snippet-end: \ No newline at end of file diff --git a/static/include/examples/run-inference/run-vision-service.go b/static/include/examples/run-inference/run-vision-service.go new file mode 100644 index 0000000000..13473e6259 --- /dev/null +++ b/static/include/examples/run-inference/run-vision-service.go @@ -0,0 +1,97 @@ +// :snippet-start: run-vision-service +package main + +import ( + "context" + "fmt" + "image/jpeg" + "bytes" + // :remove-start: + "os" + // :remove-end: + + "go.viam.com/rdk/logging" + "go.viam.com/rdk/robot/client" + "go.viam.com/rdk/services/vision" + "go.viam.com/rdk/components/camera" + "go.viam.com/rdk/utils" + "go.viam.com/utils/rpc" +) + +func main() { + apiKey := "" + apiKeyID := "" + machineAddress := "" + classifierName := "" + cameraName := "" + + // :remove-start: + apiKey = os.Getenv("VIAM_API_KEY") + apiKeyID = os.Getenv("VIAM_API_KEY_ID") + machineAddress = "auto-machine-main.pg5q3j3h95.viam.cloud" + cameraName = "camera-1" + classifierName = "classifier-1" + // :remove-end: + + logger := logging.NewDebugLogger("client") + ctx := context.Background() + + machine, err := client.New( + context.Background(), + machineAddress, + logger, + client.WithDialOptions(rpc.WithEntityCredentials( + apiKeyID, + rpc.Credentials{ + Type: rpc.CredentialsTypeAPIKey, + Payload: apiKey, + })), + ) + if err != nil { + logger.Fatal(err) + } + + // Capture image from camera + cam, err := camera.FromRobot(machine, cameraName) + if err != nil { + logger.Fatal(err) + } + + imageData, _, err := cam.Image(ctx, utils.MimeTypeJPEG, nil) + if err != nil { + logger.Fatal(err) + } + + // Convert binary data to image.Image + img, err := jpeg.Decode(bytes.NewReader(imageData)) + if err != nil { + logger.Fatal(err) + } + + // Get classifications using the image + classifier, err := vision.FromRobot(machine, classifierName) + if err != nil { + logger.Fatal(err) + } + + classifications, err := classifier.Classifications(ctx, img, 2, nil) + if err != nil { + logger.Fatal(err) + } + + // :remove-start: + if len(classifications) == 0 { + fmt.Println("No tags found") + return + } else { + for _, classification := range classifications { + fmt.Printf("Found tag: %s\n", classification.Label()) + } + } + // :remove-end: + err = machine.Close(ctx) + if err != nil { + logger.Fatal(err) + } +} +// :snippet-end: \ No newline at end of file diff --git a/static/include/examples/run-inference/run-vision-service.py b/static/include/examples/run-inference/run-vision-service.py new file mode 100644 index 0000000000..47794a3a44 --- /dev/null +++ b/static/include/examples/run-inference/run-vision-service.py @@ -0,0 +1,63 @@ +# :snippet-start: run-vision-service +import asyncio +# :remove-start: +import os +# :remove-end: +import numpy as np + +from PIL import Image +from viam.components.camera import Camera +from viam.media.utils.pil import viam_to_pil_image +from viam.robot.client import RobotClient +from viam.services.vision import VisionClient + +# Configuration constants – replace with your actual values +API_KEY = "" # API key, find or create in your organization settings +API_KEY_ID = "" # API key ID, find or create in your organization settings +MACHINE_ADDRESS = "" # the address of the machine you want to capture images from +CLASSIFIER_NAME = "" # the name of the classifier you want to use +CAMERA_NAME = "" # the name of the camera you want to capture images from + +# :remove-start: +API_KEY = os.environ["VIAM_API_KEY"] +API_KEY_ID = os.environ["VIAM_API_KEY_ID"] +MACHINE_ADDRESS = "auto-machine-main.pg5q3j3h95.viam.cloud" +CAMERA_NAME = "camera-1" +CLASSIFIER_NAME = "classifier-1" +# :remove-end: + +async def connect_machine() -> RobotClient: + """Establish a connection to the robot using the robot address.""" + machine_opts = RobotClient.Options.with_api_key( + api_key=API_KEY, + api_key_id=API_KEY_ID + ) + return await RobotClient.at_address(MACHINE_ADDRESS, machine_opts) + +async def main() -> int: + machine = await connect_machine() + + camera = Camera.from_robot(machine, CAMERA_NAME) + classifier = VisionClient.from_robot(machine, CLASSIFIER_NAME) + + # Capture image + image_frame = await camera.get_image(mime_type="image/jpeg") + + # Get tags using the ViamImage (not the PIL image) + tags = await classifier.get_classifications( + image=image_frame, image_format="image/jpeg", count=2) + + # :remove-start: + if not len(tags): + print("No tags found") + return 1 + else: + for tag in tags: + print(f"Found tag: {tag.class_name}") + # :remove-end: + await machine.close() + return 0 + +if __name__ == "__main__": + asyncio.run(main()) +# :snippet-end: \ No newline at end of file diff --git a/static/include/examples/run-inference/run-vision-service.ts b/static/include/examples/run-inference/run-vision-service.ts new file mode 100644 index 0000000000..358426226d --- /dev/null +++ b/static/include/examples/run-inference/run-vision-service.ts @@ -0,0 +1,95 @@ +// :snippet-start: run-vision-service +import { createRobotClient, RobotClient, VisionClient, CameraClient } from "@viamrobotics/sdk"; +// :remove-start: +import pkg from "@koush/wrtc"; +const { + RTCPeerConnection, + RTCSessionDescription, + RTCIceCandidate, + MediaStream, + MediaStreamTrack +} = pkg; + +// Set up global WebRTC classes +global.RTCPeerConnection = RTCPeerConnection; +global.RTCSessionDescription = RTCSessionDescription; +global.RTCIceCandidate = RTCIceCandidate; +global.MediaStream = MediaStream; +global.MediaStreamTrack = MediaStreamTrack; +// :remove-end: + +// Configuration constants – replace with your actual values +let API_KEY = ""; // API key, find or create in your organization settings +let API_KEY_ID = ""; // API key ID, find or create in your organization settings +let MACHINE_ADDRESS = ""; // the address of the machine you want to capture images from +let CLASSIFIER_NAME = ""; // the name of the classifier you want to use +let CAMERA_NAME = ""; // the name of the camera you want to capture images from +// :remove-start: +API_KEY = process.env.VIAM_API_KEY || ""; +API_KEY_ID = process.env.VIAM_API_KEY_ID || ""; +MACHINE_ADDRESS = "auto-machine-main.pg5q3j3h95.viam.cloud"; +CAMERA_NAME = "camera-1"; +CLASSIFIER_NAME = "classifier-1"; +// :remove-end: + +async function connectMachine(): Promise { + // Establish a connection to the robot using the machine address + return await createRobotClient({ + host: MACHINE_ADDRESS, + credentials: { + type: 'api-key', + payload: API_KEY, + authEntity: API_KEY_ID, + }, + signalingAddress: 'https://app.viam.com:443', + }); +} + +async function main(): Promise { + const machine = await connectMachine(); + const camera = new CameraClient(machine, CAMERA_NAME); + const classifier = new VisionClient(machine, CLASSIFIER_NAME); + + // Capture image + const imageFrame = await camera.getImage(); + + // Get tags using the image + const tags = await classifier.getClassifications( + imageFrame, + imageFrame.width ?? 0, + imageFrame.height ?? 0, + imageFrame.mimeType ?? "", + 2 + ); + + // :remove-start: + if (tags.length === 0) { + console.log("No tags found"); + return 1; + } else { + for (const tag of tags) { + console.log(`Found tag: ${tag.className}`); + } + } + + // Force exit after cleanup + process.exit(0); + // :remove-end: + return 0; +} + +// :remove-start: +// Run the script with timeout +const timeout = setTimeout(() => { + console.log("Script timed out, forcing exit"); + process.exit(1); +}, 20000); // 10 second timeout +// :remove-end: +main().catch((error) => { + // :remove-start: + clearTimeout(timeout); + // :remove-end: + console.error("Script failed:", error); + process.exit(1); +}); +// :snippet-end: \ No newline at end of file