CON-1192: Python API integrated with simulation mode #87
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The user can now create a Python enclave by creating an enclave Gradle module with a single Python script in `src/main/python` (everything else in the Gradle project stays the same). The plugin will detect this and will take a fat jar of the `python-enclave-adapter` module, the Python script, the generated Gramine manifest (which is now created dynamically based on the local environment), and will bundle them into a zip file. This zip file is what the host will scan for when it loads an enclave. Any future data that needs to be part of a Gramine enclave can be put into this zip bundle. The enclave scanning process was been rewritten to support this zip Gramine “bundle”, alongwith the existing .so GraalVM “bundle”. The Gramine enclave process will pick up this optional Python file and manually set it on the `PythonEnclaveAdapter` object. Note, the integration test currently hangs on TC though it passes if run separately. It is only testing a very basic Python enclave, but instead should be updated to use something like PyTorch.
|
There's already a new release of Jep, https://github.com/ninia/jep/releases/tag/v4.1.0! I'll look into updating it a subsequent PR. |
|
This also highlights the problem with the current approach where we take a dependency to Jep via maven (a specific version), but then install the native libs for it via apt-get. Rather than figuring out if they are compatible with each other, we should use the same source for both, which probably means that we will need to build Jep ourselves and bundle the jep .so with the SDK. |
…am to plugin test enclave
bon000
suggested changes
Nov 4, 2022
...in-enclave-gradle/src/main/kotlin/com/r3/conclave/plugin/enclave/gradle/ConclaveExtension.kt
Show resolved
Hide resolved
...adle/src/main/resources/com/r3/conclave/plugin/enclave/gradle/gramine/java.manifest.template
Outdated
Show resolved
Hide resolved
...adle/src/main/resources/com/r3/conclave/plugin/enclave/gradle/gramine/java.manifest.template
Show resolved
Hide resolved
...dle/src/main/kotlin/com/r3/conclave/plugin/enclave/gradle/gramine/GenerateGramineManifest.kt
Outdated
Show resolved
Hide resolved
|
Comments addressed, PTAL. |
bon000
suggested changes
Nov 4, 2022
...adle/src/main/resources/com/r3/conclave/plugin/enclave/gradle/gramine/java.manifest.template
Show resolved
Hide resolved
...dle/src/main/kotlin/com/r3/conclave/plugin/enclave/gradle/gramine/GenerateGramineManifest.kt
Show resolved
Hide resolved
conclave-host/src/main/kotlin/com/r3/conclave/host/internal/GramineEnclaveHandle.kt
Show resolved
Hide resolved
conclave-host/src/main/kotlin/com/r3/conclave/host/internal/GramineEnclaveHandle.kt
Show resolved
Hide resolved
integration-tests/plugin-tests/src/test/resources/test-enclave/build.gradle
Show resolved
Hide resolved
|
Pushed out another attempt to fix the integration tests. Also responded to your comments. |
|
The integration tests are now working. |
…ine, however this not a behaviour we actually want, so can be removed. But, the test shouldn't be hanging, which implies something is wrong with the Gramine integration. The test is reentract on a lock during close(), so perhaps that's where the issue might be. Either way, that's an issue for later.
…lobal timeout for all tests
…ntegration test container
… in Gramine, however this not a behaviour we actually want, so can be removed. But, the test shouldn't be hanging, which implies something is wrong with the Gramine integration. The test is reentract on a lock during close(), so perhaps that's where the issue might be. Either way, that's an issue for later." This reverts commit fd648a0.
shamsasari
force-pushed
the
shams-jep-gramine
branch
from
381beec
to
7119a2c
Compare
shamsasari
force-pushed
the
shams-jep-gramine
branch
from
f7c61b5
to
7006d37
Compare
|
I've fixed the integration tests for gramine so that now they don't hang. We get most, but not all, of the tests running. |
bon000
approved these changes
Nov 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The user can now create a Python enclave by creating an enclave Gradle module with a single Python script in
src/main/python(everything else in the Gradle project stays the same). The plugin will detect this and will take a fat jar of thepython-enclave-adaptermodule, the Python script, the generated Gramine manifest (which is now created dynamically based on the local environment), and will bundle them into a zip file. This zip file is what the host will scan for when it loads an enclave. Any future data that needs to be part of a Gramine enclave can be put into this zip bundle as well.The enclave scanning process was been rewritten to support this zip Gramine “bundle”, alongwith the existing .so GraalVM “bundle”. The Gramine enclave process will pick up this optional Python file and manually set it on the
PythonEnclaveAdapterobject.The
runtimeTypeenclave config has been made less strict and will now ignore case. For Python enclaves it will default to Gramine, rejecting any GraalVM config.A new python enclave has been added to the integration tests and which does a very basic Python test. This should be updated to something more substantial such as PyTorch. The integration tests had to be cleaned up to ensure it was running, including fixing the docker container so that Gramine can run inside it. This now means the Gramine integration tests no longer hang, but for some reason the Python test does not execute, even though it passes when run directly.