Functions that run longer than 8 seconds return empty replies.

[stealthybox]

Long running functions (8+ seconds) return with empty responses.

Expected Behavior

STDOUT should be returned in the response body.

Current Behavior

The response body is empty.

Steps to Reproduce

Deploy faas.
Create the following files:

func.yml

version: '3.1'

services:
  test:
    image: functions/alpine
    networks: [ func_functions ]
    labels:
      function: 'true'
    environment:
      read_timeout: 60
      write_timeout: 60
      fprocess: 'sh /root/func.sh'
    volumes:
    -  .:/root
networks:
  func_functions:
    external: true

func.sh

DELAY=`cat -`
echo hi, sleeping for $DELAY
sleep $DELAY
echo done sleeping, bye!

This function takes a number as input and sleeps for that duration before responding.
Let's deploy it:

> docker deploy -c func.yml issue
> docker service logs -f issue_test

And curl the service to run for different lengths:

> time curl localhost:8089/function/issue_test -XPOST -d6
hi, sleeping for 6
done sleeping, bye!

real    0m6.013s
user    0m0.004s
sys     0m0.001s

> time curl localhost:8089/function/issue_test -XPOST -d7
hi, sleeping for 7
done sleeping, bye!

real    0m7.013s
user    0m0.000s
sys     0m0.006s

> time curl localhost:8089/function/issue_test -XPOST -d8
curl: (52) Empty reply from server

real    0m8.013s
user    0m0.000s
sys     0m0.005s

> time curl localhost:8089/function/issue_test -XPOST -d9
curl: (52) Empty reply from server

real    0m9.013s
user    0m0.002s
sys     0m0.003s

The test cases for 8 and 9 seconds still return a response, but the response is empty.

Context

@cgpuglie and I are trying to run a web-scraper with nightmare-js as a faas function.
The runtime is variable, but typically longer than 8 seconds.

Your Environment

> docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:29 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:29 2017
 OS/Arch:      linux/amd64
 Experimental: true

[alexellis]

It's awesome to see you guys testing FaaS - the limit here is in the gateway rather than the watchdog which I think you've set up correctly. I think I'd consider anything over a few seconds to be asynchronous and that's something I want to tackle through the roadmap.

An asynchronous item would probably return instantly with an ID and allow polling or call back to a URL on completion.

For the time being you could get past the 8 second limit by hacking server.go in the gateway - and I think that's probably something that should be configurable.

Here's the line you'd want to edit -

https://github.com/alexellis/faas/blob/master/gateway/server.go#L65

If you run the ./build.sh file you'll get an image you can swap into your docker-compose file for the gateway called "functions/gateway:latest-dev"

Does that help?

[stealthybox]

Taking a look right now -- thanks alex!

[stealthybox]

Built and tagged it in our private registry.
Changing those values works 👍

I'll look at adding some config options for the gateway.
A pathPrefix for all of the routes would be helpful in addition to these timeouts.

[cgpuglie]

That's awesome! Thanks for the quick reply, Alex!

…

On Jun 29, 2017 4:48 PM, "leigh schrandt" ***@***.***> wrote: Built and tagged it in our private registry. Changing those values works 👍 I'll look at adding some config options for the gateway. A pathPrefix for all of the routes would be helpful in addition to these timeouts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKO3N0k7Pe0WHsaq_SMU4vG6oYjWcpqfks5sJDedgaJpZM4OJ4BC> .

[alexellis]

Great. Would be interesting to see a minimal example of the scraping contributed online somewhere or as a blog.

Btw you might find the faas-cli useful too - https://blog.alexellis.io/build-and-deploy-with-faas/

What are you thinking re: a path prefix?

[stealthybox]

I still have a draft blog-post for some linuxkit usage on windows I never finished :/

For the path Prefix stuff, I was thinking setting PathPrefix=/faas would result in

/faas/system/alert
/faas/system/functions
/faas/function/...
/faas/ui/
/faas/assets/

Not sure what the best way to do this is in gorilla/mux

[stealthybox]

I'm guessing that for the gateway config, we'd do something similar to what's implemented in watchdog with osEnv and readConfig ?

[stealthybox]

I'm not sure about the Prometheus /metrics endpoint -- can Prometheus be configured to scrape nested routes?

If you're running Prometheus in the network, it's easier to just leave it unchanged, but from outside, it could be problematic to route to.

[alexellis]

For now let's look at the timeout config in this issue. Feel free to open separate threads.

[stealthybox]

I'm confused as to why there are two levels of timeout.
Does the gateway re-invoke on bad exit codes within the gateway timeout?

If not, I'd suggest we set the request's timeout values to match the function's service spec if present.
What do you think?

[alexellis]

@stealthybox do you want to open another issue to talk about your idea re- prefixes?

[stealthybox]

Yep!
I'll do that tonight

[alexellis]

@stealthybox @cgpuglie closing this re: fix in #95 #96 - thanks for contributing and hope to see you in the Slack channel.

Did you want to raise another issue to discuss a path prefix?