[RFC] Services are all you need?

[Jack-Khuu]

(Based on conversations with @pbontrager and @DNXie)

Currently, Forge users are exposed to 2 concepts: Services (spawns processes and replicas) and ForgeActor (class doing the "work"). Should we flatten this mental model so that users only think of everything as a Service instead?

From a user's perspective, Services are a wrapper around Actor instances, with the routing setup logic intentionally abstracted away from the user. The API for spawning a Service even literally takes the underlying Actor as an argument.

https://github.com/meta-pytorch/forge/blob/4372a5490d748889070fa6374499a274185eb540/src/forge/controller/service/spawn.py#L22-L24

---

Option A: Just a rebrand

• Service => ServiceRouter
• ForgeActor => Service
• Nothing is functionally changed; just a rename
• @DNXie has started on something similar in [WIP] Renaming actor to service #124

async def spawn_service( 
     service_cfg: ServiceConfig, actor_def: Type[Service], **actor_kwargs 
 ) -> ServiceInterface: 

# Where ServiceInterface is mentally mapped as the functionality of both the Router and Service

Option B: Flatten the layers

Currently, the architecture is layered such that each class has a distinct responsibility:

• ServiceInterface wraps Service and exposes select meta APIs from it's nested child classes (actor, replica, etc)
• Service has a List[Replica] and deals with routing and load balancing
  • Recall that the final Service implementation should be an Actor itself
• Replica has ForgeActor and manages the request queueing
• ForgeActor has the actual endpoints that users write and adds Logging
• Actor is your base Monarch Actor

This looks great from a "Single Responsibility Design" aspect, but should we flatten the overall architecture to reduce this depth? Effectively moving from "instantiate a new class and adding it as a class member" to "populating a field"

Specifically, combining the interfaces of Service and ForgeAgent, call it SuperService for the sake of discussion

class SuperService(Actor):
    # All the interfaces in ForgeActor
    # These + @endpoints are the things only things users implement
    setup()
    launch()
    shutdown()

    # From Service
    call()/call_all()
    start()/stop_session()
    replica: List[Replica] # This one is a little tricky since it wraps an actor

class MyService(SuperService):
    @endpoint
    async def foo(self):
        print("a")

I haven't fully fleshed out the design/feasibility (@DNXie you should talk to @allenwang28 to flesh this out if we decide to do this)

---

Thoughts on the idea of making the idea of Service the user mental model, regardless of approach?

Are the layers of Option B unavoidable?

[DNXie]

Thanks @Jack-Khuu for writing this. Both options look reasonable to me. I can work on this once we decide on the option we wanna take. I will wait until I get more suggestions from @allenwang28 or @pbontrager.

I wonder what would this ServiceActor be in each of the plan.

[allenwang28]

Thanks @Jack-Khuu, I was thinking through this a little more. I'm not opposed to a change, but transparently I'm not sure that options A or B are the right change.

Thinking out loud:

• User writes an Actor that describes the "actions" (like generate) this construct can take
• This definition is also coupled with "how" an actor becomes replicated into an ActorMesh - this mesh becomes the fundamental unit of our service, i.e. our Replica
• The Replica should be exactly that - a replica of the other units in the service as well, making it truly stateless
• The Service that we've defined orchestrates the replicas, handles request routing / load balancing, fault tolerance and recovery, etc.

IMO this is the right layering for the service implementation, but probably the biggest challenge is the cognitive load from abstraction layers?

If I read between the lines, the fundamental problem right now is that

# User defines this...
class MyActor(Actor):
    @endpoint
    def generate(self): ...

# But actually interacts with this
service = spawn_service(MyActor, config)
await service.generate.call()  # Not actor.generate()

i.e. that the interface you write isn't the interface you use. So as you're kind of pointing out, while the abstractions are necessary, the interface feels wrong.

So A is not truly an improvement, let's focus on B. One extension of your idea could be like

class MyService(SuperService):
    # Actor-level: what happens for a single request
    @endpoint
    async def generate(self, prompt: str):
        # This runs on one actor in one replica
        return self.model.generate(prompt)
    
    # Replica-level: how actors in this replica coordinate  
    @replica_method
    async def warmup_replica(self):
        # This runs once per replica, coordinates its actors
        await self.load_model_shards()
    
    # Service-level: how replicas coordinate
    @service_method  
    async def health_check(self):
        # This runs at service level, checks all replicas
        return await self.check_all_replicas()

# Usage - same interface for everything
service = MyService(replicas=4, procs_per_replica=4)
await service.generate("hello")      # Routes to appropriate actor
await service.warmup_replica()       # Runs on all replicas  
await service.health_check()         # Service-level operation

but my issue with that is that self is no longer consistent. Based on if it's actor level, replica level or service level, the runtime contexts are legitimately different.

Some ideas could be like:

class MyActor(Actor):
    @endpoint
    async def generate(self, prompt: str):
        # self = actual actor instance
        return self.model.generate(prompt)

# But instead of spawn_service returning a separate Service object:
service = MyActor.as_service(replicas=4, procs_per_replica=4)

# Now the interface you implement IS the interface you use:
await service.generate("hello")  # Calls MyActor.generate()

or

@service(replicas=4, procs_per_replica=4)
class MyService(Actor):
    @endpoint 
    async def generate(self, prompt: str):
        return self.model.generate(prompt)

# Usage:
service = MyService()
await service.generate("hello")  # Direct call to the method you defined

or we could combine the two, and then we're stealing from ray's like:

# can define at the class level
@ray.remote(num_gpus=4)
class MyActor:
    ...

actor = MyActor.options(num_gpus=2).remote() # or at the instantiation level

how does this sound?

[DNXie]

Thanks @allenwang28 for the feedback and suggestions! I personally like the first (or the third) implementation where the user define things like replicas when they initialize the service.

What's your opinion? @pbontrager @Jack-Khuu

[allenwang28]

@DNXie I think here's what we can start with:

We can shoot for an experience like:

class Trainer(Service):
    @endpoint
    def train(self, ...):
        ...

s = Trainer.options(replicas=1, replica_shape={"procs": 4}).as_service()

Main changes I would suggest:

• I would change service.py to be orchestrator.py, also renaming the Service to Orchestrator and ServiceActor to OrchestratorActor (also re-name the top-level service as orchestrator)
• For interface.py I would change all mentions of Service to Orchestrator
• Change actor.py to service.py, rename ForgeActor to Service
• Change actors to services, for all actors there apply the Service change etc.

In Service, to enable options(...) what I might do is something like this:

@classmethod
def options(cls, replicas: int, replicas_shape: dict[str, int], ServiceConfig) -> cls:
    # basically, all services can take a whole ServiceConfig OR specify individual pieces 
    # this will return a "Class" with these options set

@classmethod
def as_service(cls, ...) -> OrchestratorInterface:
    # basically all of the functionality in `spawn_service` moved over here

[DNXie]

Thanks @allenwang28 for clarifying things. Since this is a big change, to make the changes easier for review, my plan is to split it into three PRs:

• rename services to Orchestrators
• rename actors to services
• a followup PR for the options(...).

How does it sound?

Some questions:
How about things like ServiceRequest, ServiceMetrics, and ServiceConfig?
I see the ServiceConfig is used for both Orchestrator and Actors (soon to be services), so I guess we could keep the name? ServiceMetrics and ServiceRequest, on the other hand, are specific to Orchestrator.

For interface.py I would change all mentions of Service to Orchestrator

I assume you meant to also rename ServiceEndpoint to OrchestratorEndpoint and ServiceInterface to OrchestratorInterface?