Demo troubles

[ludovicc]

Hello,

I've been trying gg, and found those issues:

1. What are the access control settings that should be used for the S3 bucket? I've been using at first 'Block public access', but got a forbidden on upload_files() operation. Then I made the bucket fully writable for the world (bad bad practice), and I could get around this issue.

gg force --jobs 100 --engine lambda src/frontend/mosh-server
→ Loading the thunks...  done (16 ms).
↗ Uploading 489 files (36.4 MiB)... terminate called after throwing an instance of 'std::runtime_error'
  what():  HTTP failure in S3Client::upload_files(): HTTP/1.1 403 Forbidden
Abandon (core dumped)

2. Compilation of mosh using gg failed with the following message:

gg force --jobs 100 --engine lambda src/frontend/mosh-server
→ Loading the thunks...  done (3 ms).
↗ Uploading 489 files (36.4 MiB)... done (2355 ms).
ld: cannot find Scrt1.o: No such file or directory
ld: cannot find crti.o: No such file or directory
ld: cannot find crtbeginS.o: No such file or directory
ld: cannot find -ltinfo
ld: cannot find -lprotobuf
ld: cannot find -lssl
ld: cannot find -lcrypto
ld: cannot find -lutil
ld: cannot find -lz
ld: cannot find -lutempter
ld: cannot find -lstdc++
ld: cannot find -lm
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find -lpthread
ld: cannot find -lc
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find crtendS.o: No such file or directory
ld: cannot find crtn.o: No such file or directory
rmdir /tmp/thunk-execute.fb0WSr: Directory not empty
std::exception
 `TZJokLWuLw23YLba.mIh.m26Qoc.AU8BG0qEvx2DyFAk00000903': process exited with failure status 1

gg-force: execution failed: TZJokLWuLw23YLba.mIh.m26Qoc.AU8BG0qEvx2DyFAk00000903

Thanks, Ludovic

[sadjad]

Hi Ludovic,

1. It's not necessary to make the bucket fully writable -- just make sure that the IAM user (the one associated with your AWS_ACCESS_KEY_ID), has AmazonS3FullAccess permission.

2. Could you please run gg describe TZJokLWuLw23YLba.mIh.m26Qoc.AU8BG0qEvx2DyFAk00000903 and post the output here, so I can take a look at the thunk that fails?

Thank you,
Sadjad

[ludovicc]

gg describe TZJokLWuLw23YLba.mIh.m26Qoc.AU8BG0qEvx2DyFAk00000903

{
 "function": {
  "hash": "VYA7BN_Oi7TEF.SFqo2yJu2fVpJOGPyeu5ThcID2g86400123508",
  "args": [
   "/__gg__/g++",
   "-L/usr/lib/gcc/x86_64-linux-gnu/7",
   "-L/usr/lib/x86_64-linux-gnu",
   "-L/usr/lib",
   "-L/lib/x86_64-linux-gnu",
   "-L/lib",
   "-L/usr/lib/x86_64-linux-gnu",
   "-L/usr/lib",
   "-L/usr/lib",
   "-L/lib",
   "-L/usr/lib",
   "-Wall",
   "-fno-strict-overflow",
   "-D_FORTIFY_SOURCE=2",
   "-fstack-protector-all",
   "-Wstack-protector",
   "--param",
   "ssp-buffer-size=1",
   "-fPIE",
   "-fno-default-inline",
   "-pipe",
   "-g",
   "-O2",
   "-pie",
   "-Wl,-z,relro",
   "-Wl,-z,now",
   "mosh-server.o",
   "../crypto/libmoshcrypto.a",
   "../network/libmoshnetwork.a",
   "../statesync/libmoshstatesync.a",
   "../terminal/libmoshterminal.a",
   "../util/libmoshutil.a",
   "../protobufs/libmoshprotos.a",
   "-lm",
   "-ltinfo",
   "-lprotobuf",
   "-pthread",
   "-lssl",
   "-lcrypto",
   "-lutil",
   "-lz",
   "-lutempter",
   "-o",
   "mosh-server",
   "-B/usr/lib/gcc/x86_64-linux-gnu/7",
   "-Wl,-rpath-link,/usr/local/lib/x86_64-linux-gnu",
   "-Wl,-rpath-link,/lib/x86_64-linux-gnu",
   "-Wl,-rpath-link,/usr/lib/x86_64-linux-gnu",
   "-Wl,-rpath-link,/usr/lib/x86_64-linux-gnu64",
   "-Wl,-rpath-link,/usr/local/lib64",
   "-Wl,-rpath-link,/lib64",
   "-Wl,-rpath-link,/usr/lib64",
   "-Wl,-rpath-link,/usr/local/lib",
   "-Wl,-rpath-link,/lib",
   "-Wl,-rpath-link,/usr/lib",
   "-Wl,-rpath-link,/usr/x86_64-linux-gnu/lib64",
   "-Wl,-rpath-link,/usr/x86_64-linux-gnu/lib"
  ],
  "envars": [
   "PATH=/__gg__",
   "GG_MANIFEST=@{GGHASH:VpwFAs0E9NnK6ue6EZs101prO_E0b2khFREMzHvw6eW0000001dd}"
  ]
 },
 "values": [
  "V0QgOTbGhxmrvSY.nijd4Qv8dsQTYlbRrO5RVZjLa1J4000b8844=../network/libmoshnetwork.a",
  "VCz5TNruI5cahljRJ3Vn7XpgTJCtxDOV_mAWbiQ3njb00001e0ea=../util/libmoshutil.a",
  "VLRlryGMuUAMW.g4MPo9A0OfahTrpQIBQgLNJLOA7e1E00029bd0=/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2",
  "VMBZ2UStz1OTCOwHbRr_GRN3zjGl9Bc9AZLdIKttYLds003e1fb6=../terminal/libmoshterminal.a",
  "VQ.jIFc5GDDsyDGF7nhfBCULpw9mM5p2u3P54o8c6Tf80012a4ec=../protobufs/libmoshprotos.a",
  "VTXmesrNEseLNA39uYrG0KuvIKH.29iPT14Yycdf1Xs80003c4b4=../crypto/libmoshcrypto.a",
  "VY8uGTaz1V.7gBy_LkiMDGdYQZbFfnLBrKdLQ4ckVMsQ000038e0=/lib/x86_64-linux-gnu/libdl.so.2",
  "VhOaEkqywgV4LiDRCVVTD2pUqVvlrYHPxbNb2tVvNGuc0019e030=mosh-server.o",
  "VpwFAs0E9NnK6ue6EZs101prO_E0b2khFREMzHvw6eW0000001dd",
  "Vva5lVMtscMO9jmybxF9W0FVsAsHAt2QSXcDIkxaNKI4000a4ef4=../statesync/libmoshstatesync.a"
 ],
 "thunks": [],
 "executables": [
  "VYA7BN_Oi7TEF.SFqo2yJu2fVpJOGPyeu5ThcID2g86400123508=/__gg__/g++",
  "VeDb5H5mtTk1vFTGivc2k7K_In2JD5Mbw_Z6VI8ftisg0022b5f8=/__gg__/ld",
  "VwfFvtNgbE1OEXyGh3w0L.U6FWKKg2hHtWrX3DR.IPKw000bca88=/__gg__/collect2"
 ],
 "outputs": [
  "output"
 ],
 "timeout": 5000
}

[siedentop]

Hi @sadjad ,

same issue here: The user as well as the GG_LAMBDA_ROLE have "AmazonS3FullAccess". Only if I make the bucket public, will the demo (building mosh) work. Otherwise, I get a 403 error as reported above.

Setup: Ubuntu 18.04 instance created on AWS, zone us-west-1.

Many thanks for this awesome project!

[siedentop]

@ludovicc , I can restrict it a little bit. If I only disable "Block public access to buckets and objects granted through new access control lists (ACLs)", then it works fine.

This did not help, but provides more details: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-execution-role-s3-bucket/

[drunksaint]

You don't need to make the bucket public. The IAM policy that I assigned to the role used by gg if it helps:

{
    "Version": "<version_date>",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:<region>:<id>:log-group:/aws/lambda/gg-lambda-function",
                "arn:aws:logs:<region>:<id>:log-group:/aws/lambda/gg-lambda-function:*"
            ]
        }
    ]
}

[drunksaint]

I believe you're getting the 403 because you have to give the List Action permission on the bucket itself, not just bucket/*.