Implement async queues

[mzohreva]

• Implement async queues
• Enable async usercalls in enclave-runner
• Limit memory usage for pending events

This is an alternative to @parthsane's PRs #194 and #217, with the following improvements:

• Has tests for the FIFO implementation
• Fixes the issue with calling async_queues usercall more than once
• Improves memory usage for pending events by removing a VecDeque that could grow unbounded

[mzohreva]

Also this does not change the ABI.

Here is an example enclave code using async queues:

#![feature(sgx_platform)]

extern crate async_queue;
extern crate fortanix_sgx_abi;

use async_queue::{
    QueueEvent, Receiver, Sender,
    Synchronizer, SynchronizationError,
};
use fortanix_sgx_abi::{FifoDescriptor, Return, Usercall};
use std::os::fortanix_sgx::usercalls::alloc::User;
use std::os::fortanix_sgx::usercalls::raw::{
    self, UsercallNrs,
    EV_RETURNQ_NOT_EMPTY, EV_USERCALLQ_NOT_FULL,
    WAIT_INDEFINITE,
};
use std::{io, mem, thread};

#[derive(Clone, Debug)]
enum QueueSynchronizer {
    Usercall,
    Return,
}

impl Synchronizer for QueueSynchronizer {
    fn wait(&self, event: QueueEvent) -> Result<(), SynchronizationError> {
        let ev = match (self, event) {
            (Self::Usercall, QueueEvent::NotEmpty) => panic!("enclave should not recv on usercall queue"),
            (Self::Return, QueueEvent::NotFull)    => panic!("enclave should not send on return queue"),
            (Self::Usercall, QueueEvent::NotFull)  => EV_USERCALLQ_NOT_FULL,
            (Self::Return, QueueEvent::NotEmpty)   => EV_RETURNQ_NOT_EMPTY,
        };
        unsafe {
            raw::wait(ev, WAIT_INDEFINITE);
        }
        Ok(())
    }

    fn notify(&self, _event: QueueEvent) {
        // any synchronous usercall would do
        unsafe {
            raw::insecure_time();
        }
    }
}

struct AsyncQueues {
    usercall_queue: FifoDescriptor<Usercall>,
    return_queue: FifoDescriptor<Return>,
}

impl AsyncQueues {
    fn new() -> io::Result<Self> {
        let usercall_queue = User::<raw::FifoDescriptor<raw::Usercall>>::uninitialized().into_raw();
        let return_queue = User::<raw::FifoDescriptor<raw::Return>>::uninitialized().into_raw();

        let r = unsafe { raw::async_queues(usercall_queue, return_queue) };
        if r != 0 {
            return Err(io::Error::from_raw_os_error(r));
        }

        let usercall_queue = unsafe {
            User::<raw::FifoDescriptor<raw::Usercall>>::from_raw(usercall_queue)
        }.to_enclave();

        let return_queue = unsafe {
            User::<raw::FifoDescriptor<raw::Return>>::from_raw(return_queue)
        }.to_enclave();

        // For some reason rust considers std::os::fortanix_sgx::usercalls::raw::FifoDescriptor
        // and fortanix_sgx_abi::FifoDescriptor different types, hence the transmutes here.
        let usercall_queue: FifoDescriptor<Usercall> = unsafe { mem::transmute(usercall_queue) };
        let return_queue: FifoDescriptor<Return> = unsafe { mem::transmute(return_queue) };

        Ok(Self { usercall_queue, return_queue })
    }

    fn sender(&self) -> Sender<Usercall, QueueSynchronizer> {
        Sender::new(self.usercall_queue.clone(), QueueSynchronizer::Usercall)
    }

    // Panics if there are existing receivers
    fn receiver(&self) -> Receiver<Return, QueueSynchronizer> {
        Receiver::new(self.return_queue.clone(), QueueSynchronizer::Return)
    }
}

fn main() {
    let queues = AsyncQueues::new().expect("failed to setup async queues");
    let usercall_queue_tx = queues.sender();
    let return_queue_rx = queues.receiver();

    const N: u64 = 100;
    let h = thread::spawn(move || {
        for i in 0..N {
            usercall_queue_tx.send(&Usercall {
                id: 1 + i,
                args: (UsercallNrs::insecure_time as _, 0, 0, 0, 0),
            }).unwrap();
        }
    });

    for _ in 0..N {
        let t = return_queue_rx.recv().unwrap();
        println!("[{:3}] {:?}", t.id, t.value);
    }
    h.join().unwrap();
}

[Goirad]

The example enclave code seems awfully verbose. Is this something that most (all?) enclaves will have to do?

[mzohreva]

The example enclave code seems awfully verbose. Is this something that most (all?) enclaves will have to do?

No, this will be abstracted away once we port MIO to SGX and users will use tokio (which relies on MIO for low level operations).

[jethrogb]

Note to self: continue reviewing at interface_sync::Receiver

[jethrogb]

github now requires me to put some text here

[jethrogb]

Note to self: continue review at struct QueueSynchronizer

[jethrogb]

Instead of the forwarding, can you do something like recv_queue.select(usercall_queue_rx.recv()) in the loop below?

[mzohreva]

That's possible, but it would complicate the already too complicated loop below. What's your main concern with having it as a separate task?

[jethrogb]

Primarily because the current flow is obfuscating what the input is to the while loop below and a lesser concern might be performance.

[mzohreva]

The input to that while loop comes from fn do_work and this task, I don't see that being obfuscated, specially because UsercallSendData has two variants. I don't think performance will be impacted much by having an extra task considering that doing select will also introduce an extra future.

We could also implement some form of back pressure on the usercall queue more easily in this form, but that might not be needed in practice.

[jethrogb]

This reorders the delivery of events compared to the current situation (events with less bits set will trigger first). I don't think that matters too much, but is there any specific reason for changing this?

[mzohreva]

The impetus for this type is that some threads may never be interested in a particular event, and therefore we could be using unbounded amount of memory storing those events. For example most enclave threads are not interested in EV_RETURNQ_NOT_EMPTY since the return queue is probably owned by just one thread, but the enclave runner needs to send that event to all threads since it does not know which thread owns that queue.

[jethrogb]

The impetus for this type is that some threads may never be interested in a particular event, and therefore we could be using unbounded amount of memory storing those events.

Makes sense.

the enclave runner needs to send [EV_RETURNQ_NOT_EMPTY] to all threads since it does not know which thread owns that queue

Well it only needs to be sent to all threads if no thread is currently waiting on that specific event:

Userspace will wake any or all threads waiting on the appropriate event when it is triggered.

(emphasis mine)

[jethrogb]

Should it just be size 0 then?

[mzohreva]

It can't be 0 (will panic) but no reason to just set it to 1 I guess.

[jethrogb]

Approving this but not yet merging as I want to make sure we publish our existing updates first.

[jethrogb]

bors r+

[bors]

Build succeeded:

• continuous-integration/travis-ci/push