Specialized worker modes

What's new

• This update introduces two new worker modes, specialized worker and backendless specialized worker, for tasklet developers. Computation backends now enjoy more flexibility with options to manage their own data transfer and process life cycle.

Other noticable features:

• Setup for connections is simplified, and Edge protocol is now used as the default. See field address in config.
• Resource wall time (grain.resource.WTime) now has an adaptive mode that infers time limit based on statistics of the first few runs.

Breaking changes

• For grain.delayed.run, arguments nolocal and rpw are deprecated in favor of local

> sha256sum grain-bridge-linux-amd64
7c185076814428e30b8ed6be3bd0a0cfa7b3734f7e7c1fd63612d698675d7267  grain-bridge-linux-amd64

UX refinement

What's new

Most of the effort for this release went into user experience improvement. Major changes include

• The default scheduler is switched to Gnaw. Integration is refined, so that it provides the same frictionless experience as the built-in scheduler, while enabling more features. For example, multiple sessions running on the same machine with the same head.listen can share a swarm of workers without further configuration.
• Edge protocol is added as a new connection discovery method. This network-filesystem-based method requires no extra setup, and is the best option in most of the scenario.
• strawman, the worker auto-scaling tool, is finally added as a Grain command.

Breaking changes

• Python 3.7 is no longer supported.

> sha256sum grain-bridge-linux-amd64
d74dcfaf17fe4ccf0d85e3e4e3bea4346375f594faaaf598ce879365d20f0ca0  grain-bridge-linux-amd64

Scalability

What's new

This release features a complete rewrite of the scheduler core in Go, with codename Gnaw. Benchmark with single worker shows 2x performance, and not to mention that it scales well in multi-worker scenarios due to its multithread concurrency. Gnaw is a standalone process that communicates with the Python frontend with Unix Domain socket. This decoupled architecture also provides an out-of-box support for running multiple missions: Multiple Python processes can connect to one Gnaw process and share the same swarm of workers. Gnaw is not enabled by default; try it out with run(..., gnaw=""). grain quit now supports quitting multiple workers with glob pattern matching.

No breaking changes are introduced.

> sha256sum grain-bridge-linux-amd64
2b794bea77e02112a8cfa56f85422ecb6f0c474a0816c098ca7e5278c4549e39  grain-bridge-linux-amd64

extensibility

What's new

This release focuses on internal refactoring instead of introducing new functionalities. Context module, the new plugin system for worker, enables worker to load custom setup-cleanup codes. The Python logging module makes logs trackable and customizable. The message serialization is now done with msgpack, making it easier to interoperate while having no changes to user experience.

Breaking changes

• Worker no longer initialize subprocess pool by default; use context module instead.
• Python 3.6 is no longer supported.

> sha1sum grain-bridge-linux-amd64
f46e425894e810688430c8a2d017d42c5a01b6c2  grain-bridge-linux-amd64

delayed and docs

What's new

Operate on pending results as if they are actual numbers, with the new frontend, Delayed.

@delayed
@subprocify
def f(i): # ith term of arctan(x)'s Taylor series, x=1
    return (1 if i%2==0 else -1) * 1/(2*i+1)

async def pi_approx(): # approximate pi as arctan(1)/4
    pi_4 = 0
    for i in range(1000):
        pi_4 += (f @ Cores(1))(i) # calculate each term on a separate core
    print(await (pi_4 * 4)) # block until the answer is ready.

Delayed comes with tutorial and documentations. Check it out!

Breaking changes

• grain unreg -> grain quit, and it will wait until all pending calculation on that worker is done. The old, forceful behavior can be used with grain quit -f
• Most of of the top-level exports are removed (e.g. grain.Cores -> grain.resource.Cores. See f2f75e4). Use their modules instead.
• Rename: grain.combine.run_combine -> grain.combine.run