Transfer environment from sandbox to workspace

[gliargovas]

As of now, export xxx does not work. We need to add support for that.

[angelhof]

The current TODOs and status of this:

• Export all variables in the sandbox and save them in a file
• Restore the variables from outside in the sandbox before executing a command
• Restore the variables from the sandbox outside with the response to wait
• Keep track of the environment and if it changes from the moment a command first executes, then we need to reexecute it. To do so we need to be able to properly parse the variable file (look in pash/compiler/expand.py for this) and keep track of changes in an efficient way (we can start by simply diffing the variable files and if they are not equal considering that there is a dependency.

I think the next step would be to do the 2nd and 3rd bullet points, and then we can think about the dependency tracking (which is also the hardest).

[gliargovas]

It seems like the Riker doesn't get exported to outside of it. Additionally, it seems like export commands and variable assignments do not appear anywhere on the trace, so it's impossible to get them without modifying either Riker or PaSh.

(1) Riker modification approach: At the end of Riker's execution we need to call something like declare -p and write the environment to a file inside the sandbox. We then need to separately resolve variable dependencies seaparately based on the declare -p outfile (that we don't yet handle).

** (2) PaSh modification approach (this seems more sound to me):** Identify the export, local and other variable assignment commands during the PaSh preprocessing time. Don't speculate these commands as handling them seems tricky. Try to replace these as soon as possible on preprocessing time. Dependency resolution could happen earlier for these (on preprocessing time), and also later. Especially for these we might not want to employ Riker (?), if safe. For other unsafe constructs we could not speculate at all for the beginning. After command expansion is supported, we could also speculate unsafe nodes.

Future notes: We also need to handle cases where a cmd result gets assigned to a variable (this is probably similar to the cmd substitution solution). Second approach could be applied only to the assignment part (after substitution) and not on the whole command (before substitution).

[gliargovas]

Especially for (2), we will need to handle the following cases of variable assignment:

• simple assignments: var=value as of now, they do not appear in the PO - fix should be simple.
• declare and typeset: especially for declare, there are several flags that might make it unsafe (either specifically handle these cases during preprocessing or handle them after command substitution is dupported?
• environment variables: export VAR=value
• read (it might not make sense to speculate this - or be extra conservative with speculation)
• command | read
• readonly
• unset

[nvasilakis]

Excellent points, @gliargovas. Quick clarification/confirmation question, what do you mean by the following?

It seems like the Riker doesn't get exported to outside of it.

[gliargovas]

Excellent points, @gliargovas. Quick clarification/confirmation question, what do you mean by the following?

It seems like the Riker doesn't get exported to outside of it.

For instance when running export foo="hello" with Riker, there is no way of getting the value of foo outside of Riker (without modifying it), as it is a separate process. We would need something like declare -p inside Riker and write Riker's environment variables to a file that we could parse just like we do with the trace file.

[ezrizhu]

Try to replace these as soon as possible on preprocessing time.

By this, do you mean catching the usage of environmental variable in pre-processing time? But what if a program internally calls a environmental variable.

[gliargovas]

Try to replace these as soon as possible on preprocessing time.

By this, do you mean catching the usage of environmental variable in pre-processing time? But what if a program internally calls a environmental variable.

Yes, exactly, we could resolve some of these (variable) dependencies statically, without Riker. This is a very good point. I don't think accessing variable values appears on the Riker trace. Maybe adding this information to the trace is necessary, so a combination of both approaches (1 and 2) is needed.

[gliargovas]

I have started changing the scheduler to support early dependency resolution. Idea for some time in the future: we could even use the infrastructure provided by the PaSh annotations to resolve dependencies statically for more cases (has to do with tier 2 cmds)

[angelhof]

Try to replace these as soon as possible on preprocessing time.

By this, do you mean catching the usage of environmental variable in pre-processing time? But what if a program internally calls a environmental variable.

I think that @ericzty is right here! We also had a meeting with @gliargovas and discussed that it because we cannot be sound in determining what variables a command uses, it makes sense to follow a conservative approach and "diff" the speculation-time and commit-time environments and if they have a difference rerun the command. This might cause unneccessary reruns but we can determine later if we can somehow optimize it IMO.

[gliargovas]

With the latest changes, only test_break seems to fail. It seems like the frontend does not send a response to the scheduler and as a result the latter waits indefinitely.