Some parameters in job_client.create() don't work

[lefnire]

I'm following the sample docs at gradient.api_sdk.clients.job_client. I can't get working_directory or job_env to work.

working_directory: auto set to /paperspace. When I pass in /app, it's ignored & /paperspace is still used (listed Console > Job > Environment; and on the Job model fetch from job_client.list()). I just cashed in and set all my Docker stuff from /app to /paperspace and it worked. Ideally I wouldn't even want to pass /app, but it'd just be picked up from WORKDIR in the Dockerfile (which is /app)

job_env: I can't find a workaround (save putting a config.json in /storage). Parameter gets gets ignored; of note, the docs has a typo:

job = job_client.create(
    ...
    job_env={
        'CUSTOM_ENV'='Some value that will be set as system environment',
    }
)

'CUSTOM_ENV'='..' should be 'CUSTOM_ENV': '..'. Not nit-picking; just that it had me second guessing my approach, since using a dict gets ignored ([j.job_env for j in job_client.list()] => [None, None, ..]) so I tried json.dumps(my_dict), still ignored. Any suggestions on getting the job_env passed in?

Overall, it seems like some params are respected, and others not; and it's hard to know which is which. Maybe a mismatch on the docs vs Github vs PyPi?

[lefnire]

Would creating /storage/config.json be a bad idea security-wise?

[mikuts]

hello @lefnire thanks for reporting this - we will take a look and push out change at the API, the env variables are getting passed along for jobs but not being returned as part of the object.

in the interim you can use experiments (gradient experiments create), which is a very similar interface as jobs with some additional syntax around /modelPath) - this will save and register your model in the paperspace system for easy deployment.

for security i wouldn't leave the /storage/config.json but set secrets at the project/cluster level and inject them in as experiment_env variables {db_read_creds: secret:DB_READ_CREDS)

[lefnire]

Thanks for the reply. The other reason I believe job_env wasn't passing is that my code cascades config.example.json -> config.json -> os.environ[x]. The 3rd was being printed as skipped in all the vars I passed. Is that we're they're accessed, as env-vars like export X=1 / os.environ['X']?

Thanks for the heads up on secrets, I'll look into that. Is there a definite security risk in /storage, like each user's /storage is publicly accessible, or is it more about good/bad practice? (Wondering if I need to do damage control now)

[mikuts]

if env variables get passed in correctly they get set the same way that you would with export X=1. you can read them in code as X = os.getenv('X'). you should see them on the actual job/experiment (which will be pushed out on the api side soon).

the config.json being on /storage is okay, just not a recommended practice, the persistent volume is only accessible to your workloads that are executed within your paperspace team. Might accidentally copy/move that config somewhere you didn't intend to, other people can see the config within the team (wouldn't want coworker seeing my github creds for instance), accidentally leak api key, ect).

[mikuts]

@lefnire re: working_dir - will update the documentation a bit to make this more clear: this is not a parameter for the WORKDIR of the container but the relative folder you want to be executing command from when you use a remote workspace (like github). shouldn't need to rebuild the container and package up your code inside every time you iterate, can instead snapshot your local directory or pull from a branch when the notebook/job/experiment/deployment executes.

[lefnire]

Thanks a million for the details @mkutsovsky !

[paperspace-philip]

@lefnire we release a new version the gradient. This version we are using envVars instead of job_env and it should work as expected.

[lefnire]

y'all rock, thanks a million!