Executor fails in some cases

[kessler-frost]

For the most part, the executor seems to be running fine but there are some cases where the workflow fails. One of these cases is:

import numpy as np

import covalent as ct
from  covalent_awsbatch_plugin.awsbatch import AWSBatchExecutor

N = 1
STALL = 1e5
X = 20

executor = AWSBatchExecutor()

@ct.electron(executor=executor)
def matrix_workload(mat_1, mat_2, stall=STALL, x=X):

    i = 0
    while i < stall:
        x * x
        i += 1

    return mat_1

@ct.lattice
def inflate(mat_1, dim=3):
    mat_2 = np.random.default_rng().integers(3, size=(dim, dim))
    return matrix_workload(mat_1, mat_2)

mat_1 = np.random.default_rng().integers(10, size=(3, 3))
dispatch_id = ct.dispatch(inflate)(mat_1)

Not sure why it fails but the status returned in the logs is FAILED.