AML-6 Model Manifest Compilation Integration

[zeryx]

This adds an additional CLI function called compile, which triggers a model_manifest.json.lock file creation, which is used by the model manifest system and data immutability & governance process.

algo compile in an algorithm directory containing a model_manifest.json file will trigger a lock operation; which calculates the md5 checksum's of all model files, and the md5 checksum of the lockfile itself; reducing the likelihood of tampering. compile requires authentication with credentials that allows access to any data API files that are defined in the manifest.

As an example; please use the model_manifest.json defined in https://gist.github.com/zeryx/5ebd1a9072a66803abff6889e051b126; and make sure that your lockfile generated contains the same md5 hashes for all relevant model files.

the ADK sibling PR is located here: algorithmiaio/algorithmia-adk-python#7

[lemonez]

see comments

[lemonez]

nice

[kennydaniel]

I think "freeze" is a fairly standard term for this action. Also the PR description doesn't match, it says algo compile

https://pip.pypa.io/en/stable/cli/pip_freeze/

[aslisabanci]

I'd also prefer to use freeze for this concept. From a software eng perspective:

lock: infers a mutual-exclusion concept; as if it's a mechanism to restrict access to files among multiple processes, as in the lockfile concept in Linux to create semaphores
compile: infers bundling/transformation of something into a runnable/interpretable package to be used by other components.

If we go by the name freeze, then for the sake of consistency we can consider renaming the variables in the ADK such as manifest_lock_path, manifest_file.json.lock, etc. too. I'll put a note in the ADK PR for this.

[lemonez]

freeze is good as well, either way.

…

On Thu, Oct 7, 2021 at 3:37 PM Kenny Daniel ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In Algorithmia/__main__.py <#113 (comment)> : > @@ -215,6 +219,9 @@ def main(): elif args.cmd == 'builds': print(CLI().getBuildLogs(args.user, args.algo, client)) + elif args.cmd == "lock": I think "freeze" is a fairly standard term for this action. Also the PR description doesn't match, it says algo compile https://pip.pypa.io/en/stable/cli/pip_freeze/ — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#113 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIVTAAB5NBGDFWNGQHJQEO3UFYOJ5ANCNFSM5FMDOZRQ> .

[aslisabanci]

I'd also prefer to use freeze for this concept. From a software eng perspective:

lock: infers a mutual-exclusion concept; as if it's a mechanism to restrict access to files among multiple processes, as in the lockfile concept in Linux to create semaphores
compile: infers bundling/transformation of something into a runnable/interpretable package to be used by other components.

If we go by the name freeze, then for the sake of consistency we can consider renaming the variables in the ADK such as manifest_lock_path, manifest_file.json.lock, etc. too. I'll put a note in the ADK PR for this.

[aslisabanci]

Just noticed that the algo_inputs variable on this line contains an extra s at the end @zeryx, so that variable becomes unused - it should be algo_input. So this is a good moment to appreciate those compiled languages once more, since they don't allow these type of programming errors😎

[lemonez]

Just noticed that the algo_inputs variable on this line contains an extra s at the end @zeryx, so that variable becomes unused - it should be algo_input. So this is a good moment to appreciate those compiled languages once more, since they don't allow these type of programming errors😎

Not to say we shouldn't appreciate compiled langs, but we could also probably turn the linting dial up a notch to catch some things too, and then we can keep appreciating Python and its versatility as well 🐍 😃