Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
e402a10
commit ac2af84
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,374 @@ | ||
| NativeLink uses a JSON file as the configuration format. This section of the | ||
| documentation will breakdown the anatomy of this configuration file with step-by-step instructions on its assembly. | ||
|
|
||
| # Assembling the Configuration File | ||
|
|
||
| Configuration JSON objects should begin with the `stores` key, followed by `workers`, `schedulers`, `servers`, and `global`. | ||
|
|
||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": [], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {} | ||
| } | ||
| ``` | ||
|
|
||
| This is the scaffolding for a NativeLink deployment configuration. | ||
|
|
||
|
|
||
| <details> | ||
| <summary>Configuring Store</summary> | ||
|
|
||
| ### Store Name | ||
|
|
||
| The value of `stores` includes top-level keys, which are the names of stores. The following example, defines the `AC_MAIN_STORE`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": { | ||
| "AC_MAIN_STORE": {} | ||
| }, | ||
| "workers": [], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Store Type | ||
|
|
||
| Once the store has been named and its object exists, | ||
| the next key is the type of store. The options are `filesystem`, `memory`, `compression`, `dedup`, `fast_slow`, `verify`, and `experimental_s3_store`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": { | ||
| "AC_MAIN_STORE": { | ||
| "filesystem": {} | ||
| } | ||
| }, | ||
| "workers": [], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Store Options | ||
|
|
||
| The contents of the object here must include `content_path`, `temp_path`, and an embedded JSON object, `eviction_policy`, which specifies the value of `max_bytes` for the store. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": { | ||
| "AC_MAIN_STORE": { | ||
| "filesystem": { | ||
| "content_path": "/tmp/nativelink/data/content_path-index", | ||
| "temp_path": "/tmp/nativelink/data/tmp_path-index", | ||
| "eviction_policy": { | ||
| // 500mb. | ||
| "max_bytes": 500000000, | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "workers": [], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
| </details> | ||
|
|
||
|
|
||
| <details> | ||
| <summary>Configuring Workers </summary> | ||
|
|
||
| ### Worker Array | ||
|
|
||
| The value of `workers` includes a top level array that embeds the worker metadata. This array always begins with the `local` object, which is the only item permitted at this time. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": [{ | ||
| "local": {} | ||
| }], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Local Worker Object Members | ||
|
|
||
| The Local object has five components, `worker_api_endpoint`, `cas_fast_slow_store`, `upload_action_results`,`work_directory`, and `platform_properties`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": [{ | ||
| "local": { | ||
| "worker_api_endpoint": { | ||
| "uri": "grpc://127.0.0.1:50061", | ||
| }, | ||
| "cas_fast_slow_store": "WORKER_FAST_SLOW_STORE", | ||
| "upload_action_result": { | ||
| "ac_store": "AC_MAIN_STORE", | ||
| }, | ||
| "work_directory": "/tmp/nativelink/work", | ||
| "platform_properties": { | ||
| "cpu_count": { | ||
| "values": ["16"], | ||
| }, | ||
| "memory_kb": { | ||
| "values": ["500000"], | ||
| }, | ||
| "network_kbps": { | ||
| "values": ["100000"], | ||
| }, | ||
| "cpu_arch": { | ||
| "values": ["x86_64"], | ||
| }, | ||
| "OSFamily": { | ||
| "values": [""] | ||
| }, | ||
| "container-image": { | ||
| "values": [""] | ||
| }, | ||
| } | ||
| } | ||
| }], | ||
| "schedulers": {}, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
| </details> | ||
|
|
||
|
|
||
| <details> | ||
| <summary>Configuring Schedulers </summary> | ||
|
|
||
| ### Scheduler Name | ||
|
|
||
| The value of `schedulers` includes top-level keys, which are the names of stores. The following example, defines the `MAIN_SCHEDULER`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": { | ||
| "MAIN_SCHEDULER": {} | ||
| }, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Scheduler Type | ||
|
|
||
| Once the scheduler has been named and its object exists, | ||
| the next key is the type of scheduler. The options are `simple`, `action_scheduler`, `grpc_scheduler`, `property_modifier_scheduler`, and `worker_scheduler`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": { | ||
| "MAIN_SCHEDULER": { | ||
| "simple": {} | ||
| }, | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Scheduler Options | ||
|
|
||
| The contents of the scheduler type object defines the options. For a list of options see the documentation. There is an example below. | ||
|
Check failure on line 191 in nativelink-config/examples/example-nl-config.md
|
||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": { | ||
| "MAIN_SCHEDULER": { | ||
| "simple": { | ||
| "supported_platform_properties": { | ||
| "cpu_count": "minimum", | ||
| "memory_kb": "minimum", | ||
| "network_kbps": "minimum", | ||
| "disk_read_iops": "minimum", | ||
| "disk_read_bps": "minimum", | ||
| "disk_write_iops": "minimum", | ||
| "disk_write_bps": "minimum", | ||
| "shm_size": "minimum", | ||
| "gpu_count": "minimum", | ||
| "gpu_model": "exact", | ||
| "cpu_vendor": "exact", | ||
| "cpu_arch": "exact", | ||
| "cpu_model": "exact", | ||
| "kernel_version": "exact", | ||
| "OSFamily": "priority", | ||
| "container-image": "priority", | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "servers": [], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| </details> | ||
|
|
||
| <details> | ||
| <summary>Configuring Servers</summary> | ||
|
|
||
| ### Servers | ||
|
|
||
| The `servers` configuration object is an array, with two objects, `public`, and `private_workers_servers`. | ||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": {}, | ||
| "servers": [{ | ||
| "name": "public" | ||
| },{ | ||
| "name": "private_workers_servers" | ||
| }], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Public Server | ||
|
|
||
| The `public` server consists of a `listener` object and a `services` object. The `listener` object is simple and includes an `http` with a `socket address`. The `services` server consists of a `cas`, an `ac`, the `execution`, `capabilities`, and `bytestream`. | ||
|
Check failure on line 251 in nativelink-config/examples/example-nl-config.md
|
||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": {}, | ||
| "servers": [{ | ||
| "name": "public", | ||
| "listener": { | ||
| "http": { | ||
| "socket_address": "0.0.0.0:50051" | ||
| } | ||
| }, | ||
| "services": { | ||
| "cas": { | ||
| "main": { | ||
| "cas_store": "WORKER_FAST_SLOW_STORE" | ||
| } | ||
| }, | ||
| "ac": { | ||
| "main": { | ||
| "ac_store": "AC_MAIN_STORE" | ||
| } | ||
| }, | ||
| "execution": { | ||
| "main": { | ||
| "cas_store": "WORKER_FAST_SLOW_STORE", | ||
| "scheduler": "MAIN_SCHEDULER", | ||
| } | ||
| }, | ||
| "capabilities": { | ||
| "main": { | ||
| "remote_execution": { | ||
| "scheduler": "MAIN_SCHEDULER", | ||
| } | ||
| } | ||
| }, | ||
| "bytestream": { | ||
| "cas_stores": { | ||
| "main": "WORKER_FAST_SLOW_STORE", | ||
| } | ||
| } | ||
| }, | ||
| },{ | ||
| "name": "private_workers_servers" | ||
| }], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| ### Private Server | ||
|
|
||
| The `private` server consists of a `listener` object and a `services` object. The `listener` object is simple and includes an `http` with a `socket address`. The `services` server consists of an `experimental_prometheus` object with a `path` field, a `worker_api` object with `scheduler_field`, and an `admin` object. | ||
|
Check failure on line 304 in nativelink-config/examples/example-nl-config.md
|
||
|
|
||
| ```json | ||
| { | ||
| "stores": {}, | ||
| "workers": {}, | ||
| "schedulers": {}, | ||
| "servers": [{ | ||
| "name": "public", | ||
| "listener": { | ||
| "http": { | ||
| "socket_address": "0.0.0.0:50051" | ||
| } | ||
| }, | ||
| "services": { | ||
| "cas": { | ||
| "main": { | ||
| "cas_store": "WORKER_FAST_SLOW_STORE" | ||
| } | ||
| }, | ||
| "ac": { | ||
| "main": { | ||
| "ac_store": "AC_MAIN_STORE" | ||
| } | ||
| }, | ||
| "execution": { | ||
| "main": { | ||
| "cas_store": "WORKER_FAST_SLOW_STORE", | ||
| "scheduler": "MAIN_SCHEDULER", | ||
| } | ||
| }, | ||
| "capabilities": { | ||
| "main": { | ||
| "remote_execution": { | ||
| "scheduler": "MAIN_SCHEDULER", | ||
| } | ||
| } | ||
| }, | ||
| "bytestream": { | ||
| "cas_stores": { | ||
| "main": "WORKER_FAST_SLOW_STORE", | ||
| } | ||
| } | ||
| }, | ||
| },{ | ||
| "name": "private_workers_servers", | ||
| "listener": { | ||
| "http": { | ||
| "socket_address": "0.0.0.0:50061" | ||
| } | ||
| }, | ||
| "services": { | ||
| "experimental_prometheus": { | ||
| "path": "/metrics" | ||
| }, | ||
| // Note: This should be served on a different port, because it has | ||
| // a different permission set than the other services. | ||
| // In other words, this service is a backend api. The ones above | ||
| // are a frontend api. | ||
| "worker_api": { | ||
| "scheduler": "MAIN_SCHEDULER", | ||
| }, | ||
| "admin": {} | ||
| } | ||
| }], | ||
| "global": {}, | ||
| } | ||
| ``` | ||
|
|
||
| *`global`* is simple and cna just be added at the end as the configuration object for file descriptors. | ||
|
Check failure on line 373 in nativelink-config/examples/example-nl-config.md
|
||
| </details> | ||
