Viam - Iterative Development

[danic85]

As part of the development process, we would usually expect to be able to iterate and test code quickly on the Pi platform as dependencies exist there that do not on other platforms like Ubuntu.

The use of Mutagen.io allows us to sync code from a local development environment to the Pi filesystem, and then we can use the Viam server to connect and configure the robot over the web. (SSH and VNC are also an option, but not preferred).
Viam Processes can be created to execute code when the Viam server starts, and can reference modules.
For reusable code, Viam Modules can be created and either configured locally or released on the Viam Registry.

I am looking to get a better understanding of the development process using Viam. Below are my assumptions and questions:

• Creating a new script is simple and the Code sample tab can be used as a template.
• The Code sample adapts based on the configuration of your robot to include examples of using modules you have configured
• There are three variables included in the sample code that must be set in order to communicate with the specific Viam server: api_key, api_key_id and the RobotClient.ataddress() first argument.
• In some example scripts, these values are set using the command os.getenv (for example robot_api_key = os.getenv('ROBOT_API_KEY') or ''. We understand that these should be set as shown, is there a simpler approach if the server and script are on the same device?
• The example scripts rely heavily on asyncio. It is an assumption that this is a preferred approach to allow concurrent execution of modules.
• Logging can be achieved with the from viam.logging import getLogger module and this will surface output to the Logs tab. What other approaches are recommended to debug when a log output does not explain the cause of the issue?
• Assuming a main.py file is used, we can configure this as a Viam Process. This will then run whenever the Viam Server executes and will restart if it fails. If any changes are made we can either restart the Viam server via the UI (or command line), or kill the specific process on the device, in which case Viam server will restart it. Is this the recommended approach to testing small changes to the code?
• The guide for creating a local process recommended installation of dependencies in the same directory, this fixed my issue but fills the directory with the dependencies. Is there a cleaner way of achieving this? (script and config for reference)
• It appears that restarting the Viam server triggers a reinstall of the dependencies of each configured module on a fresh virtual environment. It was mentioned that a different module configuration approach could be used that created a dotfile to only allow installation once. Can you provide an example of this, and is there then a way to trigger a full reinstall if needed without SSHing onto the device to remove the dotfiles?
• Being able suppress logs from a module besides filtering would be great. For example the viam:camera:csi-pi module outputs a lot of errors to the log that are warnings from the library, it would be good to ignore these.
• Once the script is running and we want to exit we can configure __del__() handlers to deconstruct any instances and reset hardware to the initial configuration (as required).

@mcvella a bit of a brain dump this time, but hopefully helpful to clarify assumptions if you wouldn't mind reviewing. Don't worry about responding to each point, if the assumption is correct it can be ignored. There are some questions peppered in there too.

I'll do another for module specific questions, I need to refresh my memory.

[mcvella]

See responses inline. I skipped the ones where I think what you are saying is a statement that I agree with.

• The Code sample adapts based on the configuration of your robot to include examples of using modules you have configured
  Yes, but two things:

1. In Viam parlance, "resources" are components or services. Modules can actually contain/provide more than one resource; they are a packaging and distribution mechanism. So I'd replace "modules" with "resources" in your statement.
2. There are some exceptions to what gets generated in a code sample. For example, resources with generic or custom APIs currently cannot have meaningful API calls generated in the sample code.

• In some example scripts, these values are set using the command os.getenv (for example robot_api_key = os.getenv('ROBOT_API_KEY') or ''. We understand that these should be set as shown, is there a simpler approach if the server and script are on the same device?

For processes, not currently. For modules, these can either be configured via Viam configuration for that resource or env variables can be configured as documented here (https://docs.viam.com/registry/configure/#use-environment-variables-with-a-registry-module).

• The example scripts rely heavily on asyncio. It is an assumption that this is a preferred approach to allow concurrent execution of modules.
  For our Python SDK, most of the SDK methods expect that you are working with async client code.
  Instances of resources provided by modules will run concurrently in their own process. Let me know if there are more specifics you are interested in on this.

• Logging can be achieved with the from viam.logging import getLogger module and this will surface output to the Logs tab. What other approaches are recommended to debug when a log output does not explain the cause of the issue?

I am not sure I have a specific answer other than to leverage language-specific features like stack traces. If you have a specific example of where you've had issues with debugging let me know and we can try to find solutions together.

• Assuming a main.py file is used, we can configure this as a Viam Process. This will then run whenever the Viam Server executes and will restart if it fails. If any changes are made we can either restart the Viam server via the UI (or command line), or kill the specific process on the device, in which case Viam server will restart it. Is this the recommended approach to testing small changes to the code?

Yes, for now. We've discussed adding a product feature to make this easier for any resource. Now that you know our Jira is public, feel free to open a ticket (or I can, but we do want user feedback directly and that's one reason we have a public Jira!)

• The guide for creating a local process recommended installation of dependencies in the same directory, this fixed my issue but fills the directory with the dependencies. Is there a cleaner way of achieving this? (script and config for reference)
• It appears that restarting the Viam server triggers a reinstall of the dependencies of each configured module on a fresh virtual environment. It was mentioned that a different module configuration approach could be used that created a dotfile to only allow installation once. Can you provide an example of this, and is there then a way to trigger a full reinstall if needed without SSHing onto the device to remove the dotfiles?

Check out the example here of using a venv and only doing the install once: https://github.com/viam-labs/detection-dock/blob/main/run.sh

This works well with modules since a new version install would wipe out the directory thus triggering a new install. With processes, you'd need to manually delete the .installed file if there were new deps/a new version of the script.

• Being able suppress logs from a module besides filtering would be great. For example the viam:camera:csi-pi module outputs a lot of errors to the log that are warnings from the library, it would be good to ignore these.

Noted! Again, feel free to open a Jira ticket or let me know and I can.

• Once the script is running and we want to exit we can configure __del__() handlers to deconstruct any instances and reset hardware to the initial configuration (as required).

This works with modules, I don't think processes have lifecycle methods like this. You could use signal handlers: https://www.instructables.com/Learn-How-to-Capture-and-Handle-OS-Signals-Like-SI/

I believe most of what you're looking to build is reusable code, can you give me a few examples of code you'd choose to use processes over modules for? I think these use cases would be interesting for our product team as well.

[danic85]

I believe most of what you're looking to build is reusable code, can you give me a few examples of code you'd choose to use processes over modules for? I think these use cases would be interesting for our product team as well.

The only use case I have for a process at the moment is the execution of the main.py file that imports the other resources and configures the inter-resource communication (pubsub topics). This is how that looks, although I will be able to refactor it significantly with the introduction of Viam resources.

The goal is that when the server starts/stops the robot starts/stops, so that I don't need to configure that manually on the board. That combined with the self-contained architecture of the viam modules should make things a lot cleaner and easier to configure & customise.

I think the next step for me is to see what I can use from the registry in place of the current resources in this framework, then create Viam modules for the resources that don't currently exist. It will take a while but I'll tag you with questions (and the occasional code review to make sure i'm on the right track) as I go if that works for you.

If you have any suggestions for modules to replace the current list we discussed in #72 feel free to send over some links.

Thanks!

Edit: adding tag - @mcvella

[danic85]

Another thing I forgot - do you have examples of how to add controls to a module? I saw a few pages about using the control tab but I may have missed docs around building the controls themselves.
I'm thinking of adding a manual interface for the serial connection (so free text entry and submission), or buttons and toggles for various other inputs.
It would also be good to output the state of modules visually on the control tab, so anything related to that would be great.

@mcvella

[mcvella]

So, if you are using a built-in API, then you would get a card in the control tab for an instance of that resource. However, for non-builtins you'd have to test with one of our SDKs. Generic is built-in, and there is a DoCommand card that you can use from the control tab to test your DoCommands from there.

When you say state of the component, do you mean if it is up and running?

[danic85]

So we don't have any way of adding custom UI to components in the control tab? I'm thinking if I wanted to have different controls like 'sit', 'stand', etc for debugging or demo, access to a folder to view images stored by the camera, single button presses that fire events to trigger behaviour. I can think of a few other use cases where greater customisation of the control tab would be useful.

For the state, I mean surfacing what is going on within the robot via the UI, so everything from 'a pin is reading high' to 'the robot is in face detection mode rather than object detection'. Currently I read the logs and interpret, but a more user friendly interface would be really valuable.

@mcvella

[mcvella]

No current way to add custom UIs to the control tab, correct. I think its a great idea and I shared it with Eng. The hard part I think would be how it would be bundled and made available to the Viam app. For now, you can of course create your own typescript app for custom things.

Also interesting on state - today resources can maintain their own state, but how that is maintained and exposed is fairly arbitrary. For example, motors have an isRunning() method that the control tab uses, but not all resources would have a similar state access method. Something to think through and probably related to custom UIs - perhaps one could specify which state method(s) to access and at what frequency....