Issues with encrypted files with GSC

[quertenmont]

Following up on the discussion (with @dimakuv ) started in gsc issues (but nicely closed by @mkow )
gramineproject/gsc#110 (comment)

My gsc manifest is the following

loader.pal_internal_mem_size = "256M"
loader.insecure__use_host_env = true
libos.check_invalid_pointers = false
loader.env.KMP_AFFINITY = "granularity=fine,noverbose,compact,1,0"
sgx.remote_attestation = "dcap"
sgx.enclave_size = "1G"
sgx.preheat_enclave = true
sgx.thread_num = 8

#sgx.allowed_files = [ "file:/resources/" ]   ### REMARK, IF I USE THIS INSTEAD OF FS.MOUNTS IT WORKS

fs.mounts = [
  { type = "encrypted", path = "/resources", uri = "/resources", key_name = "_sgx_mrsigner" },
]

but my container fails to starts with the following error

Gramine is starting. Parsing TOML manifest file, this may take some time...
Detected a huge manifest, preallocating 64MB of internal memory.
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - loader.insecure__use_host_env = true       (forwarding environment vars from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

[P1:T1:] error: error mounting "/resources" (encrypted) under /resources: -22
[P1:T1:] error: Error during libos_init() in init_mount (-22)

@dimakuv suggested that my volume (mounted by docker on /resources) might not be attached to the container and that is the cause of this error. But I am 100% sure the volume is properly mounted and is accessible because when instead of using fs.mounts I use sgx.allowed_files, I can start my gscfied container and see my volume is properly mounted and usable (with file persistence in it).

So I am opened to other idea and suggestion on how to solve this.

Thanks in advance for helping,
Loic

[mkow]

Are the files you have in /resources/ encrypted by Gramine encryption tool? Also, you can try setting log level to debug in the manifest, maybe you'll get a more detailed error message.
And more importantly, please read https://gramine.readthedocs.io/en/latest/manifest-syntax.html#encrypted-files, if you haven't.

[quertenmont]

As I tried to explain in the ticket above,
I would like to have /resources/ being encrypted by Gramine,
but I am not able to have my gsc docker image running when I enable file mount with encryption or even file mount at all.

I am only able to mount the directory using
sgx.allowed_files = [ "file:/resources/" ]
which has no file encryption at all.

Here is an extract of the gramine debug log whn trying to mount the directory using

fs.mounts = [
  { type = "encrypted", path = "/resources", uri = "/resources", key_name = "_sgx_mrsigner" },
]

The log:

warning: PalVirtualMemoryProtect is unimplemented in Linux-SGX PAL
debug: Gramine was built from commit: e18bc05b17fd704b259cb0401f928dc4ec5199a6
debug: Host: Linux-SGX
debug: LibOS xsave_enabled 1, xsave_size 0xa80(2688), xsave_features 0xe7
debug: Initial VMA region 0x3cbf4000-0x3ccd0000 (LibOS) bookkeeped
debug: Initial VMA region 0x3ecc1000-0x40000000 (manifest) bookkeeped
debug: ASLR top address adjusted to 0xb1df000
debug: host is Linux-SGX and remote attestation type is 'dcap', adding SGX-specific /dev/attestation/ files: report, quote, etc.
debug: LibOS loaded at 0x3cbf4000, ready to initialize
debug: mounting "file:/" (chroot) under /
debug: mounting "proc" (pseudo) under /proc
debug: mounting "dev" (pseudo) under /dev
debug: mounting "dev:tty" (chroot) under /dev/tty
debug: mounting "sys" (pseudo) under /sys
debug: Creating pipe: pipe.srv:cd4eb7a6cf0a205b56f0d46b1fe5e31c437629c6ff386ed4bea1277efa9cf8b9
[P1:T1:] debug: mounting "/resources" (encrypted) under /resources
[P1:T1:] error: error mounting "/resources" (encrypted) under /resources: -22
[P1:T1:] error: Error during libos_init() in init_mount (-22)
debug: PalProcessExit: Returning exit code 1

this is not giving me much more information... but maybe it does for you ?

I don't know if my issue is related to the fact that I am using GSC and not gramine straight,
of if the issue is related to the kubernetes volume definition ?

My k8s pod configuration for azure aks has the form:
(I tried with both an emptyDir volume or a volume claim mounting a CSI azure disk volume)

spec:
  containers:  
   - name: main
     volumeMounts:
    - mountPath: /resources/
      name: resources-volume
    - mountPath: /var/run/aesmd
      name: var-run-aesmd
  volumes:
#  - name: resources-volume
#    persistentVolumeClaim:
#      claimName: resources-volume
  - emptyDir: {}
    name: resources-volume
  - hostPath:
      path: /var/run/aesmd
      type: ""
    name: var-run-aesmd

Thanks for helping
Loic

[mythi]

My k8s pod configuration for azure aks has the form:

where in your pod spec you speficy the the SGX resource requests? sgx.intel.com/epc: ....

[quertenmont]

@mythi in case it wasn't clear, it's a partial k8s configuration that I sent, just to highlight what's going on with the volume.
I can send the full k8s configuration if you wish, but I doubt it's going to help.

Did any of you has already tried to run on aks with gsc and encrypted files on a aks volume ?

[mythi]

@mythi in case it wasn't clear, it's a partial k8s configuration that I sent, just to highlight what's going on with the volume. I can send the full k8s configuration if you wish, but I doubt it's going to help.

OK, it wasn't that obvious but thanks for the clarification.

Did any of you has already tried to run on aks with gsc and encrypted files on a aks volume ?

FWIW, I've succesfully used what used to be Gramine protected files from k8s hostPath and emptyDir mounts but I haven't yet tried the encrypted files. I can give it a try on my side and let you know if I can reproduce.

[quertenmont]

@mythi If you can try and share your setup, It would be very helpful to me :-)

[mythi]

{ type = "encrypted", path = "/resources", uri = "/resources", key_name = "_sgx_mrsigner" },

try: uri = "file:/resources"

[quertenmont]

@mythi THANKS!!!!!!!!!

I can't believe that it was so stupid....

Can I say that error reporting could be improved ?
Anyway, thank you very much, really appreciated.

Loic

[mythi]

Can I say that error reporting could be improved ? Anyway, thank you very much, really appreciated.

-22 (invalid argument) helped me to realize the mount itself had invalid params but this case made me also think should this "syntax error" be detected during manifest parsing. Another observation is that the manifest syntax documentation is not very clear on this file: prefix requirement for URI mount paths.

[dimakuv]

Well, that's a fail on our side... The error message could definitely be more clear. Sorry. @mkow @oshogbo Could anyone maybe add a better error message in such cases (missing file:)? And what happens at all in this case, why there is no error message?

[oshogbo]

Let me look into that.