Designs the PAL API v4 changes

[jiazhang0]

This issue is used to address the definition of PAL API v4 for enclave-cc.

[YangLiang3]

I will provide more details further.

[jiazhang0]

@YangLiang3 Most of it looks good. But why not call pal_create_process as pal_exec?

[mythi]

@YangLiang3 Most of it looks good. But why not call pal_create_process as pal_exec?

Do I read the table correct: rune exec was mapped to separate pal_create_process and pal_exec in v3 and now the proposal is that it is enough to have just one pal_create_process call? Is this path different in enclave-cc that justifies the change?

[jiazhang0]

@ying2liu @bigdata-memory @qzheng527 Please ACK whether this is the formal matrix Gramine and Occlum will adapt to.

[YangLiang3]

@YangLiang3 Most of it looks good. But why not call pal_create_process as pal_exec?

Do I read the table correct: rune exec was mapped to separate pal_create_process and pal_exec in v3 and now the proposal is that it is enough to have just one pal_create_process call? Is this path different in enclave-cc that justifies the change?

hi Mikko, the change is due to the fact occlum and gramine have the requirements about combining the pal_create_process and pal_exec as one. So we make some change to it. pal_create_process name often confuse us, so I will suggest occlum team makes a change about it.

[YangLiang3]

@YangLiang3 Most of it looks good. But why not call pal_create_process as pal_exec?

Occlum NGO version had combined the pal_create_process and pal_exec and called it as pal_create_process.

[bigdata-memory]

A question about the Pal_create_process, from LibOS perspective, it has no any knowledge about the calling sequence from CRI, so when Pal_create_process gets called, how LibOS-side can know the Pal_init has already been invoked ? or it just simply doesn't care about it by design ? thanks.

[ying2liu]

Even though naming doesn't impact Gramine PAL API implementation, I still think pal_exec is a better name.

[jiazhang0]

A question about the Pal_create_process, from LibOS perspective, it has no any knowledge about the calling sequence from CRI, so when Pal_create_process gets called, how LibOS-side can know the Pal_init has already been invoked ? or it just simply doesn't care about it by design ? thanks.

Essentially, PAL API is called by untrusted part. So the possible unordered calls to PAL API would happen and the trusted part should keep its internal state machine to prevent from the impact or even security issue caused by unordered calls to PAL API.

[YangLiang3]

A question about the Pal_create_process, from LibOS perspective, it has no any knowledge about the calling sequence from CRI, so when Pal_create_process gets called, how LibOS-side can know the Pal_init has already been invoked ? or it just simply doesn't care about it by design ? thanks.

hi Gordan, the libos will mange the state machine. For the occlum's view, if there is no pal_init, pal_exec can't find the enclave id, error will happen.

[guzongmin]

My understanding is: When we are talking about the PAL API, actually we are talking about the LibOS wrapper/adapter interface name. The enclave CC would develop adapters for every LibOS or any detail implementation. So the rough design is used to make sure the sequence is good enough for all target LibOS.

I think we should not only list the API names, the more important thing is to list all the API's behavior and the parameters.

[YangLiang3]

pal_create_process

My understanding is: When we are talking about the PAL API, actually we are talking about the LibOS wrapper/adapter interface name. The enclave CC would develop adapters for every LibOS or any detail implementation. So the rough design is used to make sure the sequence is good enough for all target LibOS.

I think we should not only list the API names, the more important thing is to list all the API's behavior and the parameters.

hi zongmin, the adaptor for every libos is unified, so we don't need to expose different adaptors layer. Unifying interface name is the step 1. And step 2 will be detailed parameters and API behavior. I need to confirm the API's parameter and API behavior reasonable, then expose them today.

[bigdata-memory]

Another question about pal_get_evidence, from the perspective of Runelet, without something like get_status how it is possible to detect the status of enclave and the process running inside it ? For example, the Gramine cannot generate report or quote if the running process doesn't provide the user data, or even provided, the timing is unpredictable to Runelet. so the pal_get_evidence would be blocked or never return if it is a sync callback.

[YangLiang3]

Enclave Runtime Programming Guide V4

1. Background

The enclave runtime currently supported by runE are occlum and WAMR (WebAssembly Micro Runtime). In order to facilitate other libos programs to run in runE, a set of enclave runtime API interfaces is defined. Libos only needs to support this set of API interfaces to run as an enclave runtime in runE.

2. enclave runtime in runE

runE enclave runtime is bounded by the enclave runtime pal API layer, below the API layer is runE, above the API layer is the enclave runtime, and the operating mode is libos.

2.1 enclave runtime pal API definition

struct pal_attr_t {
    const char*     args;
    const char*     log_level;
    char* user_priv;
};

struct stdio_fds {
    int stdin, stdout, stderr;
};

struct pal_exec_args {
    char *path;
    char *argv[];
    char *env[];
    struct stdio_fds *stdio;
    int pid;
    int *exit_value;
}__attribute__((packed));

struct pal_evidence_args {
    int  ev_type; 
    bool result;
    struct la {
        char *target_info;
        char *report;
     };
     struct ra {
        char *quote;
        int  quote_len;
    };
}

struct pal_kill_args {
    int pid;
    int sig;
}__attribute__((packed));

struct pal_opt {
    int pal_version();
    int pal_init(struct pal_attr_t *attr);
    int pal_exec(struct pal_exec_args *args);
    int pal_get_evidence(struct pal_evidence_args *args);
    int pal_kill(struct pal_kill_args *args);
    int pal_destroy();
};

2.2 enclave runtime Library file naming and function naming rules

The enclave runtime is generated as a so dynamic library, which is dynamically loaded by rune using dlopen; the enclave runtime needs to export symbols according to the function named in the previous chapter.

3. pal interface

3.1 pal_get_version()

Description

Indicate PAL API version number implemented by runelet and enclave runtime; runelet is compatible with any enclave runtimes equal to or less than the indicated value. If this symbol is undefined in enclave runtime, version 1 is assuemd by runelet.

The value of this global variable is the version of pal api, refer to the implementation:

int pal_get_version()
{
    return 4;
}

Prototype

int pal_get_version();

Parameters

N/A

Return value

@int: the PAL API version of the current enclave runtime.

3.2 pal_init

The main task of this interface should be to create an enclave space and complete the memory layout of the enclave space; libos also needs to complete the initialization of components such as VM, FS, and NET. Reference implementation:

int pal_init(const struct pal_attr_t *attr)
{
    ...
    sgx_launch_token_t token;
    get_token(&token);
    sgx_create_enclave(..., token, ...);
    ...
}

Description

Do libos initialization according to the incoming attr parameters.

Prototype

struct pal_attr_t {
	const char *args;
        const char *log_level;
        char *user_priv;
};

int pal_init(struct pal_attr_t *attr);

Parameters

@args: Pass the required parameters of libos (can be instance path etc.).
@log_level: Log level.
@user_priv: User private parameter.

Return value

0: Success
-EINVAL: Invalid argument
-ENOSYS: The function is not supported

3.3 pal_exec

The main job of this interface is to run application in enclave, reference implementation:

int pal_exec(struct pal_exec_args *args)
{
    ...
    args->pid = libos_create_process(...);
    libos_exec(args->pid);
    ...
}

Description

run the user application.

Prototype

struct pal_stdio_fds {
	int stdin, stdout, stderr;
};

struct pal_exec_args {
	char *path;
	char *argv[];
	char *env[];
	struct pal_stdio_fds *stdio;
	int  pid;
        int *exit_value;
}__attribute__((packed));

int pal_exec(struct pal_exec_args *args);

Parameters

@path: The path of the binary file to be run (relative path in the libos file system).
@argv: Binary parameters, ending with a null element.
@env: Binary environment variables, ending with a null element.
@stdio: The fd of stdio.
@pid: If the value is 0, pid stores the pid of the new process in libos.
@exit_value: Exit value of the process

Return value

0: Success
-EINVAL: Invalid argument
-ENOSYS: The function is not supported

3.4 pal_get_evidence()

according to the input parameter, retrieving local attestation or remote attestation's evidence.

int pal_get_evidence(struct pal_evidence_args *args)
{
    ...
    caseLA:
        args-> la = libos_verify_local_report(...);
    case RA:
        args -> ra = libos_get_ra_quote(...);
    ...
}

Description

If ev->type is LA, then complete LA verification, la->target_info is null, return -ENOSYS. If ev->type is RA, then get the application enclave's quote info.

Prototype

struct pal_evidence_args {
    int  ev_type; 
    bool result;
    struct la {
        char *target_info;
        char *report;
     };
     struct ra {
        char *quote;
        int  quote_len;
    };
}

Parameters

@ev_type: 0 indicates local attestation; 1 indicates remote attestation;
@result: If LA, it indicates the LA is success or not; If RA, it indicates the quote info from App enclave
@la:  la -> target_info, agent enclave's target info value; la -> report indicates application's report info.
@ra: ra -> quote, application enclave's quote info

Return value

0: Success
-EINVAL: Invalid argument
-ENOSYS: The function is not supported

3.5 pal_kill

The main job of this interface is to send a signal to the specified pid, refer to the implementation:

int pal_kill(int pid, int sig)
{
    ...
    libos_kill(...)
    ...
}

Description

Send signals to processes running in enclave runtime.

Prototype

int pal_kill(int pid, int sig);

Parameters

@pid: Send to all processes if equal to -1, or send to current process if equal to 0, or send to the process that owns the pid if others.  
@sig: Signal number to be sent.

Return value

0: Success
-EINVAL: Invalid argument
-ENOSYS: The function is not supported

3.6 pal_destroy

The main job of this interface is to destroy the entire enclave space. If it is libos, you need to do component de-initialization before destroying the enclave. Reference implementation:

int pal_destroy(void) {
    ...
    libos_uninitialize();
    sgx_destroy_enclave(global_eid);
    ...
}

Description

Destroy libos instance.

Prototype

int pal_destroy();

Parameters

N/A

Return value

0: Success
-ENOSYS: The function is not supported

[YangLiang3]

Another question about pal_get_evidence, from the perspective of Runelet, without something like get_status how it is possible to detect the status of enclave and the process running inside it ? For example, the Gramine cannot generate report or quote if the running process doesn't provide the user data, or even provided, the timing is unpredictable to Runelet. so the pal_get_evidence would be blocked or never return if it is a sync callback.

thanks gordan's feedback, for gramine, it's hard to implement this function. Let's align it in a meeting

[mythi]

I think we should not only list the API names, the more important thing is to list all the API's behavior and the parameters.

I'd also want to see the proposed changes documented like this:

"v3 is not enough for enclave-cc because we need: ..."

[YangLiang3]

I think we should not only list the API names, the more important thing is to list all the API's behavior and the parameters.

I'd also want to see the proposed changes documented like this:

"v3 is not enough for enclave-cc because we need: ..."

In fact, there are the following changes related to V3

1. Combine pal_create_process and pal_exec as one
2. Add one parameter "user_priv" which can be used by libos for specific data
3. Add rune attest's requirement including Remote attestation and Local Attestation(Maybe this interface is not easy to implement. If it's not reasonable after discussion, we can remove it)

[mythi]

Add rune attest's requirement including Remote attestation and Local Attestation

In enclave-cc, where's rune attest with remote attestation used? My understanding was that this would be part of attestation-agent with an SGX specific KBC.

[YangLiang3]

Add rune attest's requirement including Remote attestation and Local Attestation

In enclave-cc, where's rune attest with remote attestation used? My understanding was that this would be part of attestation-agent with an SGX specific KBC.

rune is not bound to Agent enclave or Attestation-Agent. Rune is a independent runtime

[mythi]

rune is not bound to Agent enclave or Attestation-Agent. Rune is a independent runtime

I know but this issue is to discuss PAL changes needed by agent enclave / enclave-cc.

[ariel-adam]

@jiazhang0 is this issue still relevant or can be closed?
If it's still relevant to what release do you think we should map it to (mid-November, end-December, mid-February etc...)?

[mythi]

@jiazhang0 is this issue still relevant or can be closed?

@jiazhang0 I'd think we could close this for now and re-open later if any PAL API changes become relevant. Agree?

[dcmiddle]

cc: @ying2liu @bigdata-memory

[bigdata-memory]

Agree to close it for now and re-open it if needed, thanks.

[hairongchen]

closed for now, will reopen if needed