[Bug]: Can not log into dev environment when ~/.ssh/environment file becomes too large

[james-boydell]

Steps to reproduce

This is the same bug as #2257.

Actual behaviour

No response

Expected behaviour

No response

dstack version

0.18.40

Server logs

Additional information

It appears it's not fixed and the environment file is still being appended to, occasionally resulting in failed ssh connections.

One of our users has only been using dstack for two weeks and has already encountered this bug.

They have noted they get 3-4 ungraceful shutdowns a week. Maybe this is contributing?

[peterschmidt85]

cc @un-def

[un-def]

Sorry for the delayed response. In theory, 10 ungraceful shutdowns are enough to break ~/.ssh/environment if there are 100+ env variables in the run configuration (Docker image ENV directives, env run property, and DSTACK_* variables combined).

What is the reason for such shutdowns?

[james-boydell]

@un-def Exact reasons are unknown as the logging doesn't provide exact information. However it's typically either a spot interruption or out of memory.

[peterschmidt85]

@un-def BTW, I suppose dstack should clearly show the reason, right? At least for spot interruption or out of memory. COuld we also check that our statuses pass this information?

[un-def]

Spot interruption is INTERRUPTED_BY_NO_CAPACITY: https://dstack.ai/docs/guides/troubleshooting/#cause-1-spot-interruption

OOM is CONTAINER_EXITED_WITH_ERROR with 137 exit status

[james-boydell]

Just to comment on exit status, when using AWS, we get INTERRUPTED_BY_NO_CAPACITY when we OOM.