support maverick commands #202
Comments
|
Note that The same kind of thing happens with Given that, it might be better for python/-api to use systemd directly using pythonic modules: |
|
Also a lot of the maverick services are controlled through puppet, ie through localconf.json. So as well as starting/stopping services, they should also be set in localconf.json to persist the change. Or maybe that's not always wanted.
|
|
re interacting with systemd directly, that was my plan where possible. Last time I had a look at options this one was at the top of my list: https://github.com/facebookincubator/pystemd |
|
Ooh interesting. Can we guarantee all the platforms will have dbus? |
|
Also note that |
|
from pystemd.systemd1 import Unit
import glob
glob.glob('/etc/systemd/system/maverick*')
unit = Unit(b'maverick-webdev.service')
unit.load()
unit.Service.GetProcesses()
unit.Unit.Names unit.Unit.ActiveState unit.Unit.SubState unit.Service.GetProcesses() unit.Unit.Stop('replace') unit.Unit.ActiveState unit.Unit.SubState unit.Unit.Start('replace') hmmmmnot sure if this is worth it... we have to run -api as sudo and we cant enable / disable services... just stick to subprocess... |
|
Interesting OK, we'll hit this problem with whatever we use to control systemd. There are ways we can provide sufficient granular root access to a process. |
|
I like the idea of a microservice here. I'll put something together. Preferences on the communication? Rest, graphql, something else? |
|
If we used dbus, we could use the permissions manager in that instead, which is more flexible: But then that means depending on dbus, which isn't always a given in non-desktop environments. Probably not a good idea. |
|
Looks like systemd requires dbus.. One option is to add a PolicyKit pkla: This allows us to start/stop/restart (any) services as the mav user: Unfortunately it doesn't allow us to specify which services can be controlled (allows all system services), and also doesn't allow us to enable/disable services, which requires root filesystem access (to change the symlinks in /etc/systemd/system/multi-user.target.wants/): But it's an option - the enable/disable could be done separately through |
|
Another option is to create user systemd manifests, by putting them in ~/.config/systemd/user. This is possibly a cleaner way of doing it, although might be a bit more confusing for people used to looking for them in /etc/systemd/system. This allows control through the mav user without any escalated privileges, however not sure if it allows processes to be run as root, which some of the mav processes do. |
|
Using subprocess and just calling |
|
refactored the async process runner here: https://github.com/goodrobots/maverick-api/blob/7ef29f4283e618320710c825c3b21513ab62bb96/maverick_api/modules/base/util/process_runner.py For services we only care about the return value which is returned by the |
|
Still need to add parsing of the service structure but queries, mutations and subscriptions in place. |
|
@fnoop Setup: I have not been able to find a way to attach callbacks to systemd. The following finds and then polls each of the import time
from pystemd.systemd1 import Manager
class ServiceStatus:
__slots__ = ["_name", "_running", "_enabled", "update_time", "callback"]
def __init__(self, name, callback = None):
self._name = name
self._running = b"unknown"
self._enabled = b"unknown"
self.callback = callback
self.update_time = time.time()
@property
def name(self):
return self._name[:self._name.find(b".service")].decode()
@property
def running(self):
if self._running == b'running':
return True
elif self._running == b'unknown':
return None
else:
return False
@property
def enabled(self):
if self._enabled == b'enabled':
return True
elif self._enabled == b'unknown':
return None
else:
return False
def update_running(self, running = b'unknown'):
if self._running == running:
pass
else:
self._running = running
self.update_time = time.time()
if self.callback:
self.callback(self)
def update_enabled(self, enabled = b'unknown'):
if self._enabled == enabled:
pass
else:
self._enabled = enabled
self.update_time = time.time()
if self.callback:
self.callback(self)
def __eq__(self, other):
if not isinstance(other, ServiceStatus):
return False
return self._name == other._name and self._running == other._running and self._enabled == other._enabled
def __hash__(self):
return hash((self._name, self._running, self._enabled))
def list_units():
with Manager() as manager:
print("Version", manager.Manager.Version)
print("Architecture", manager.Manager.Architecture)
services = {}
while 1:
time.sleep(1) # backoff the systemd calls to spare CPU cycles
for unit_path, enabled_status in manager.Manager.ListUnitFilesByPatterns(["enabled", "disabled"], ["maverick-*"]):
time.sleep(0.2)
idx = unit_path.find(b"maverick-")
if not idx:
continue
sevice_name = unit_path[idx:]
if sevice_name not in services:
ss = ServiceStatus(sevice_name, callback=some_callback)
ss.update_enabled(enabled=enabled_status)
services[sevice_name] = ss
continue
services[sevice_name].update_enabled(enabled=enabled_status)
for unit in manager.Manager.ListUnitsByNames(services.keys()):
time.sleep(0.2)
services[unit[0]].update_running(running=unit[4])
def some_callback(service):
print(service.name)
print(f"running:{service.running}")
print(f"enabled:{service.enabled}")
print()
list_units()Any thoughts? |
|
What the above does not do is get the status of |
|
Do the callbacks in https://github.com/facebookincubator/pystemd/blob/master/examples/monitor_all_units_from_signal.py do us for start/stop service? Does it work for indirect services (apparently called 'template units')? I would be less bothered about enabled/disabled, as we can set that in config and depend on a maverick run to commit. It would be great to bypass puppet if we can do it directly of course, but doesn't have to be a top priority. |
|
Yes they do pick up the changes in start / stop of template units... This is what I have at the moment: import pystemd
import asyncio
import tornado.ioloop
import functools
from pprint import pprint
from pystemd.dbuslib import DBus
from pystemd.systemd1 import Unit
unit_data = [
("maverick-visiond.service", "visiond"),
("maverick-api@dev.service", "api@dev"),
]
def process(msg, error=None, userdata=None):
msg.process_reply(True)
print("*" * 80)
# print(" * New message:\n")
# pprint({k: v for k, v in msg.headers.items() if v is not None})
member = msg.headers.get("Member", None)
if member == b"PropertiesChanged":
# pprint(msg.body[1])
unit_path = msg.headers.get("Path", None)
sub_state = msg.body[1].get(b"SubState", None)
timestamp_monotonic = msg.body[1].get(b"StateChangeTimestampMonotonic", None)
if unit_path and sub_state:
if unit_path in userdata:
print(userdata[unit_path]["name"])
print(sub_state)
print(timestamp_monotonic)
elif member == b"UnitFilesChanged":
print("An enable / disable event occured")
else:
pass
print("#" * 80)
print("\n")
def read_callback(fd, event, **kwargs):
kwargs["bus"].process()
async def monitor(event, units):
loop = tornado.ioloop.IOLoop.current()
with DBus() as bus:
for unit_data in units.values():
bus.match_signal(
unit_data["unit"].destination,
unit_data["unit"].path,
b"org.freedesktop.DBus.Properties",
b"PropertiesChanged",
process, # callback for this message
units,
)
bus.match_signal(
b"org.freedesktop.systemd1",
None,
None,
b"UnitFilesChanged",
process, # callback for this message
units,
)
loop.add_handler(
bus.get_fd(), functools.partial(read_callback, bus=bus), loop.READ
)
await event.wait()
loop = tornado.ioloop.IOLoop.current()
units = {}
for unit_service_name, name in unit_data:
unit = Unit(unit_service_name)
units[unit.path] = {"unit": unit, "name": name}
event = asyncio.Event()
loop.add_callback(functools.partial(monitor, event, units))
loop.start()we can use the existing code to populate the following with all the services we care about unit_data = [
("maverick-visiond.service", "visiond"),
("maverick-api@dev.service", "api@dev"),
]This will allow us to update the running state of services without making subprocess calls. The code above also detects that something has been enabled / disabled in the system, but as I don't have a nice way of knowing what service was changed, we need to check all of the services that we are interested in using the existing subprocess call. At least then the subprocess calls are limited to a change event (rather than a dumb loop) |
|
Great, that's really cool! So we just need to work out how to automatically populate unit_data somehow. systemd/psystemd must have a way to query a list of all services? |
|
Currently I'm parsing the directory structure that the maverick status command uses to determine what services to show ~/software/maverick/bin/status.d/ . If we ignore that I can get all the maverick- services using the Manager interface, but it doesn't list instances of template units e.g. it gives maverick-api@.service but doesn't give us dev or fc. So we could have maverick update the directory structure when a user adds a sitl/ros/mavros/-api instance... Or keep trying to get them via pystemd |
|
okay, we can get all the maverick- services with the following code def read_services_dbus(self):
services = []
with Manager() as manager:
for _unit in manager.Manager.ListUnits():
idx = _unit[0].find(b"maverick-")
if idx == -1:
continue
services.append({
"unit": Unit(_unit[0]),
"category_name": None,
"category_display_name": None,
"command": _unit[0][0:-8].decode(),
"service_name": _unit[0][9:-8].decode(),
"service_display_name": _unit[1].decode(),
"category_index": None,
"service_index": None,
"last_update": None,
"enabled": None,
"running": True if _unit[4] == b'running' else False,
})
for service in services:
application_log.debug(f"Adding service: {service}")I'm re-writing the services module to use the dbus code. I wont have time to work on it much this week. |
|
dbus code is in... https://github.com/goodrobots/maverick-api/blob/master/maverick_api/modules/api/maverick/maverick_service.py note that the hot reload of the api server causes the dbus code to stop watching for events... need to add custom logic to unload this module |
|
need to support systems without dbus |
|
What system is that? When I did a bit of research it appeared that dbus is basically a dependency of systemd, except in unusual circumstances. |
|
windows subsystem for linux (I use it for testing) |
|
Ah never ever considered running it on windows :) |
|
There is a development configuration option that needs to be set to True, otherwise graphiql returns a 403 |
|
Stopping (or starting) a single service a single time, we get 2 subscription updates:
|
|
That's because I'm treating all dbus statuses that are not 'running' as 'dead'. Transitional states between dead an running will cause a subscription update but the service will be reported as dead.
|
|
What are the transitional states? If we can't see them in the message and they're not useful to the front end then ideally we shouldn't see them, just the final change to live, or a final change to dead. |
|
I can't recall the exact names but they are effectively reporting starting and stopping states. My concern was that unless the service is in that final state of running, for all purposes it's still dead and hence I was reporting it as such. |
control services
maverick start ..maverick stop ..maverick status ..The text was updated successfully, but these errors were encountered: