GSC conversion for REST API

[ADL-work]

Description of the problem

Unable to convert Flask server docker container using GSC

Steps to reproduce

Hi,

I am trying to do a simple conversion of docker images running Flask server with REST API using GSC but fail, reproduced steps as below

app.py

from flask import Flask, jsonify, request
app=Flask(__name__)
#we are importing our function from the colors.py file

@app.route("/",methods=['GET','POST'])
def index():
    if request.method=='GET':
#getting the url argument       
        url = request.args.get('url')
        result={'Result': "GET result"}
        return jsonify(result)
    else:
        return jsonify({'Error':"This is a GET API method"})
if __name__ == '__main__':
    app.run(debug=True,host='0.0.0.0', port=5000)

ubuntu18.04-flask.dockerfile

# Sets the base image for subsequent instructions
FROM ubuntu:18.04
# FROM python:3.7-slim-stretch
# Sets the working directory in the container  
WORKDIR /app
RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install -y python3
RUN apt-get install -y python3-distutils python3-pip python3-apt

# Copies the dependency files to the working directory
COPY requirements.txt /app/requirements.txt
# Install dependencies
RUN pip3 install -r requirements.txt
# Copies everything to the working directory
COPY . /app
RUN chmod 777 ./app.py
# Command to run on container start    
CMD [ "python3" , "./app.py" ]

requirements.txt

click==7.1.2
Flask==1.1.2
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
Werkzeug==1.0.1

Build the docker image
docker build -t ubuntu18.04-flask-demo -f ./ubuntu18.04-flask.dockerfile .

Convert the build (using generic manifest)
./gsc build --insecure-args ubuntu18.04-flask-demo test/generic.manifest

./gsc sign-image ubuntu18.04-flask-demo enclave-key.pem

Run the converted docker
docker run -d --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket -p 5000:5000 gsc-ubuntu18.04-flask-demo

Expected results

Calling at localhost should return response
curl localhost:5000

{
  "Result": "GET result"
}

Actual results

The docker container hanging with these logs only

Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
 * Debugger is active!
 * Debugger PIN: 138-003-840

Additional information

• SGX is installed in the VM (Azure DC sv3)

Questions:

• Can I use some pre-configured docker images (like python:3.7-slim-stretch) or it has to be ubuntu:18.04 and then install python3 in the docker image?
• Is it possible to run a docker container (with Flask server) and also provide arguments in the docker run command ? something like this:
  docker run -p 5000:5000 -it --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-<some-converted-image> '$ARG'
• Do you have some working examples with Python Flask server (or any RESTFUL API) I can refer to?

Thanks a lot

[dimakuv]

• Can I use some pre-configured docker images (like python:3.7-slim-stretch) or it has to be ubuntu:18.04 and then install python3 in the docker image?

Yes you can. You can use any Ubuntu-based Docker image as your base image in the ubuntu18.04-flask.dockerfile file. The only requirement is that Gramine's base image (specified in config.yaml of GSC) is compatible with your Dockerfile's base image.

• Is it possible to run a docker container (with Flask server) and also provide arguments in the docker run command ? something like this:
  docker run -p 5000:5000 -it --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-<some-converted-image> '$ARG'

Yes. For this, you need to use the --insecure-args option during the gsc build step. Which you already do. So it should just work.

• Do you have some working examples with Python Flask server (or any RESTFUL API) I can refer to?

Unfortunately no. But I don't see any specific problems with REST API or with Flask -- I mean, I don't think your problem is about network connectivity. Your problem is most probably with some manifest option that is missing from your minimal test/generic.manifest.

So the next step for you:

1. Please show us your config.yaml GSC file.
2. Please tell us which GSC commit you're using (cd gsc && git show --summary).
3. Please try to re-build your application with gsc build --debug, to get debug information. This will force GSC and Gramine to print a lot of info. You can analyze this log yourself and find the potential issues, or you can attach the log here, so we can take a look.

[ADL-work]

Thanks @dimakuv , please see my responses in the step below

my test/generic.manifest

# Some workloads like Python may generate huge manifest files
loader.pal_internal_mem_size = "128M"

sgx.enclave_size = "4G"
sgx.thread_num = 8

sgx.trusted_files = [
  "file:/gramine/app_files/entrypoint.manifest",  # unused entry, only to test merging of manifests
]

So the next step for you:

1. Please show us your config.yaml GSC file.

# Specify the OS distro. Currently tested distros are
# ``ubuntu:18.04``, ``ubuntu:20.04``, ``ubuntu:21.04`` and ``centos:8``.
Distro: "ubuntu:18.04"

# If the image has a specific registry, define it here.
# Empty by default; example value: "registry.access.redhat.com/ubi8".
Registry: ""

# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though
Gramine:
    Repository: "https://github.com/gramineproject/gramine.git"
    Branch:     "master"

# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
#   - legacy out-of-tree driver: use something like the below values, but adjust the branch name
#         Repository: "https://github.com/01org/linux-sgx-driver.git"
#         Branch:     "sgx_driver_1.9"
#
#   - DCAP out-of-tree driver: use something like the below values
#         Repository: "https://github.com/intel/SGXDataCenterAttestationPrimitives.git"
#         Branch:     "DCAP_1.11 && cp -r driver/linux/* ."
#
#   - DCAP in-kernel driver: use empty values like below
#         Repository: ""
#         Branch:     ""
#
SGXDriver:
    Repository: ""
    Branch:     ""

2. Please tell us which GSC commit you're using (cd gsc && git show --summary).
   commit 72b296d5272863aea5200d4cda5b9e240d08151c (HEAD -> master, origin/master, origin/HEAD)

3. Please try to re-build your application with gsc build --debug, to get debug information. This will force GSC and Gramine to print a lot of info. You can analyze this log yourself and find the potential issues, or you can attach the log here, so we can take a look.

I made with the debug and received some warnings like this:

{"log":"[P1:T1:python3] trace: ---- openat(AT_FDCWD, \"/usr/lib/python3.6/__pycache__/warnings.cpython-36.pyc.1638345408\", O_WRONLY|O_CREAT|O_EXCL|0x80000, 0644) = -13\n","stream":"stderr","time":"2022-07-13T09:39:14.239605538Z"}
{"log":"warning: Disallowing access to file '/usr/lib/python3.6/__pycache__/shutil.cpython-36.pyc.1635719976'; file is not trusted or allowed.\n","stream":"stderr","time":"2022-07-13T09:39:14.246526475Z"}
{"log":"warning: Disallowing access to file '/usr/lib/python3.6/__pycache__/bz2.cpython-36.pyc.1638721720'; file is not trusted or allowed.\n","stream":"stderr","time":"2022-07-13T09:39:14.250241313Z"}
{"log":"warning: Disallowing access to file '/usr/lib/python3.6/__pycache__/_compression.cpython-36.pyc.1639286240'; file is not trusted or allowed.\n","stream":"stderr","time":"2022-07-13T09:39:14.253179997Z"}

I suspect that there must be more allow files/folders need to be added in the manifest file ?

Additionally, would it be possible to add the gsc to main path so that I can run it anywhere instead of inside gsc folder? Perhaps a soft link would work?

Thank you for your suggestions

Here's the log file
7f5a0178ef612c072784e479e51839cbe153bb49c8ebfbe150109f87469f7838-json.log

[dimakuv]

Additionally, would it be possible to add the gsc to main path so that I can run it anywhere instead of inside gsc folder? Perhaps a soft link would work?

You can just copy it yourself, if you want to. Something like ln -sf gsc /usr/local/bin should work. We didn't implement the installation procedure, because nobody seemed to care (until your request now :)).

Here's the log file

The log file has this as the last syscall executed:
pselect6(0, 0, 0, 0, 0x4aa11bc0, 0) ...

This is the way to sleep for some time (we can't see for how much time because we only see the pointer to the "time struct", which is at address 0x4aa11bc0). Are you sure your program is not just sleeping for like several minutes?

Other than that, I didn't find anything special in the log...

[ADL-work]

This is the way to sleep for some time (we can't see for how much time because we only see the pointer to the "time struct", which is at address 0x4aa11bc0). Are you sure your program is not just sleeping for like several minutes?

That's kind of strange, as my python program given above didn't make any sleep time, or maybe the Flask library does that without my understanding.

Anyways, should I include all necessary python libraries in the trusted files of the manifest, just to make sure it can read/write the libraries ?

Thank you

[dimakuv]

Anyways, should I include all necessary python libraries in the trusted files of the manifest, just to make sure it can read/write the libraries ?

This actually shouldn't be necessary, as GSC does it by itself.

If you think that reading/writing some files (libraries) can be a root cause, then you can try sgx.file_check_policy = "allow_all_but_log" manifest option.

[ADL-work]

I've tried with the given manifest option, but it was the same result, the app hanging but not receiving GET request from the curl command.

Also recently, when I tried to build new gsc images, there's an error appears:

Program protoc-c found: YES (/usr/bin/protoc-c)
Program protoc found: NO
meson.build:262:4: ERROR: Program 'protoc' not found or not executable
A full log can be found at /gramine/build/meson-logs/meson-log.txt
NOTICE: You are using Python 3.6 which is EOL. Starting with v0.62.0, Meson will require Python 3.7 or newer
Failed to build unsigned graminized docker image gsc-python3.7-flask-demo-unsigned.

Do you have any suggestion for this ? (I'm running local python3 version at 3.6.9)
Thank you
Update: I've installed 3.7 and update alternative python3 to 3.7 and re-installed all needed pip3 libs, but the issue is still there.

[dimakuv]

Please update your GSC to the latest version. We recently made a change to core Gramine and to GSC.

[llly]

I tried app.run(debug=False,host='0.0.0.0', port=5000) without debugger.
It can work.

Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [29/Jul/2022 08:05:12] "GET / HTTP/1.1" 200 -

$ curl localhost:5000
{"Result":"GET result"}

[ADL-work]

I tried app.run(debug=False,host='0.0.0.0', port=5000) without debugger. It can work.

Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [29/Jul/2022 08:05:12] "GET / HTTP/1.1" 200 -

$ curl localhost:5000
{"Result":"GET result"}

Hi @llly ,
Yes it seems removing the debugging option made the Flask server code running (at least with the GET).
Thank you for the suggestion!

[dimakuv]

Since it seems to work, I will close this issue.