gg force --engine lambda failing after emptying bucket

[drunksaint]

Something seems to have broken after I emptied the s3 bucket that was being used by gg. I deleted the .gg directory and re-initialized gg. Still it seems to be wanting to download some old thunks. Running gg force --engine lambda isn't adding any thunks to the bucket now. Is there any other cache that I should empty? How do I go about debugging this?

$ gg infer filetest i.txt o.txt 
$ gg force --engine lambda o.txt 
→ Loading the thunks...  done (0 ms).
No files to upload.
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\nterminate called recursively\n  what():  "}
[warning] operational failure: TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160': HTTP/1.1 404 Not Found\nterminate called recursively\n"}
[warning] operational failure: TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160': HTTP/1.1 404 Not Found\n"}
$ gg force o.txt 
→ Loading the thunks...  done (0 ms).
$ cat o.txt
3

[sadjad]

Removing ~/.cache/gg/remotes and all its content should fix the issue. gg will reupload the missing files.

If you look at the error message, it says that the blobs VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016 and VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160 are missing from the bucket. In fact, gg didn't try to upload those file because it has done that before (according to a cache entry in ~/.cache/gg/remotes) and it's not aware that you emptied the bucket!

This reminds me we were meant to create a gg bucket-sync or a similar tool to resync the local index and the remote bucket, but never got around to it.

[drunksaint]

that fixed the issue! Now I'm able to see the thunks in the bucket. thanks!

I'm running another script and this time it is giving me a different error:

$ gg force --engine lambda output.txt
→ Loading the thunks...  done (0 ms).
No files to upload.
gg-force: no execution engine could execute T_iyxy6SOlvutbNC9MrsSTZH8sNaxcqEZyICDFtu0aWk000002b4
$ gg force output.txt
→ Loading the thunks...  done (0 ms).
$

Can I see the execution logs somewhere? Or is there some other way to debug this?

[sadjad]

This is probably because the total size of the input files to that thunk exceeds 230 MiB. That's a hard-coded limit in the lambda engine, since Lambdas come with 500 MiB of disk space (gg reserve the rest of the space for the output).

You have a couple options: either change your thunk to use smaller files as input, or, change the code and increase the maximum limit. Just be careful of the total 500 MiB disk space and the fact that your output needs to fit on disk as well.

[sadjad]

When running a gg job, it's possible that some thunks exceed the limitations of the engine. In that case you can run gg force like this:

gg force --engine lambda --fallback-engine local thunk_name

If none of the engines listed by --engine can run the thunk, it will be run locally (of course, it will need to copy all of the dependencies to the local machine). Of course, in your case here, I think you have a single thunk that you wanna run, so, this may not be so useful.

[drunksaint]

Oh, my script + files were 266MB XD. Changing the limit fixed the issue :). The fallback option will be quite useful later i think!

Looks like the lambda workers don't seem to have libpython3.7m.so.1.0 for nuitka to work with:

$ gg infer argtest 65 i.txt o.txt --arg 23 --inputfile j.txt --outputfile p.txt
$ gg force --engine lambda o.txt 
→ Loading the thunks...  done (0 ms).
No files to upload.
argtest: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
std::exception
 `TMDchyW8zrDeCv8W2XKHvrTClA8mMZQzoZxo_941Odv0000001b5': process exited with failure status 127

gg-force: execution failed: TMDchyW8zrDeCv8W2XKHvrTClA8mMZQzoZxo_941Odv0000001b5

Will see if they have some something else I can use. I'll take a look at this tomorrow.

[drunksaint]

@sadjad can gg be told to upload the libpython.so file as well?

[sadjad]

Oh, that's the issue with Nuitka: it relies on the availability of Python libraries on the host. Let me check something first.

[sadjad]

I think for now the quickest solution might be going back to PyInstaller. There's a way to have gg to include the required libraries, such as libpython.so, but it's a little painful. Meanwhile, I'm gonna try to figure out if there's way to use Nuitka with Lambda.

[drunksaint]

Cool, I'll try the pyinstaller route tomorrow. Thanks for looking at Nuitka!

[drunksaint]

Looks like running a pyinstaller binary on lambda also has a similar problem:

[7] Error loading Python lib '/tmp/_MEIovTa5m/libpython3.7m.so.1.0': dlopen: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/_MEIovTa5m/libpython3.7m.so.1.0)

I'm starting a new thread [#48] for this since this thread refers to the solved empty bucket issue.