Got 404 error for /+search on AWS Fargate deployment

[myrodgkim]

I have created a simple Docker image for devpi-server. using attached Dockerfile
When running on my local machine, it works fine.

But, when I deploy the docker image to AWS Fargate, I got some errors.
For instance, when I accessing /+search , server respond with 404 error. for simple GET request,
but AJAX /+search seems working.

predicate mismatch for view search (content type = application/json)

I have searched above message, I think it is related to web framework, but I don't know about devpi's framework.
(I am just using it by installing devpi, I don't know how to trace the error)

And, sometimes I got 502 response from nginx. I guess devpi python process has died or can't respond to nginx.

The docker file that I attached doesn't contain additional configurations.
but I will add some additional configurations for AWS Fargate instance
Fargate instance will have .5 vCPU and 1 GB Memory. (I am not sure about how much does it needs)
I will add AWS LB in front of Fargate for HTTPS.
I will add EFS shared filesystem as Index storage.
I will add devpi-ldap for authentication.

First, I suspected X-outside-url, the devpi process can't access ALB Frontend URL. I don't know this affects the result.
Second, I tested without nginx using default 3141 port, but I also got 404 errors. I have tested simple curl.

curl -v fargate_ip/+search

[myrodgkim]

Dockerfile

FROM python:3.7

RUN apt update && apt install -y nginx

# install devpi server
RUN pip install devpi-server devpi-web devpi-client devpi-semantic-ui

# configuration parameters for devpi server
ENV DEVPI_THEME semantic-ui
ENV DEVPI_PORT 3141
ENV DEVPISERVER_SERVERDIR /devpi

# prepare server
COPY ./entrypoint.sh /

# configure nginx for proxy
# RUN apt update && apt install -y nginx
# COPY ./nginx/etc/nginx/conf.d/backend.conf /etc/nginx/conf.d/
COPY ./nginx/etc/nginx/sites-enabled/default /etc/nginx/sites-enabled/

# for devpi direct
EXPOSE $DEVPI_PORT

# for nginx
EXPOSE 80

# run server
CMD ["/entrypoint.sh"]

entrypoint.sh

#!/usr/bin/env bash
set -e

[[ -f $DEVPISERVER_SERVERDIR/.serverversion ]] || initialize=yes

# Properly shutdown devpi server
shutdown() {
    devpi-server --stop  # Kill server
    kill -SIGTERM $TAIL_PID  # Kill log tailing
}

trap shutdown SIGTERM SIGINT

if [ -f $DEVPISERVER_SERVERDIR/.serverversion ]; then
  echo "Using existing directory"
else
  echo "Initializing devpi for the first time"
  devpi-init
fi

# Need $DEVPISERVER_SERVERDIR
devpi-server --start \
--host 0.0.0.0 \
--port $DEVPI_PORT \
--theme $DEVPI_THEME \
--debug

# run nginx web server
nginx

DEVPI_LOGS=$DEVPISERVER_SERVERDIR/.xproc/devpi-server/xprocess.log

tail -f /var/log/nginx/access.log &
tail -f /var/log/nginx/error.log &
tail -f $DEVPI_LOGS &
TAIL_PID=$!

# Wait until tail is killed
wait $TAIL_PID

# Set proper exit code
wait $DEVPI_PID
EXIT_STATUS=$?

nginx/etc/nginx/sites-enabled/default

server {
    server_name localhost $hostname "";   
    listen 80;
    gzip             on;
    gzip_min_length  2000;
    gzip_proxied     any;
    gzip_types       application/json; 

    proxy_read_timeout 60s;
    client_max_body_size 64M;

    # set to where your devpi-server state is on the filesystem
    root /data;  

    # try serving static files directly
    location ~ /\+f/ {
        # workaround to pass non-GET/HEAD requests through to the named location below
        error_page 418 = @proxy_to_app;
        if ($request_method !~ (GET)|(HEAD)) {
            return 418; 
        }

        expires max;
        try_files /+files$uri @proxy_to_app;
    }
    # try serving docs directly
    location ~ /\+doc/ {
        # if the --documentation-path option of devpi-web is used,
        # then the root must be set accordingly here
        root /data;
        try_files $uri @proxy_to_app;
    }
    location / {
        # workaround to pass all requests to / through to the named location below
        error_page 418 = @proxy_to_app;
        return 418;
    }
    location @proxy_to_app {
        proxy_pass http://localhost:3141;
        # proxy_set_header X-outside-url $scheme://$http_host;
        set $prot 'https';
        if ($host = 'localhost') {
            set $prot 'http';
        }
        proxy_set_header X-outside-url $prot://$host;
        proxy_set_header X-Real-IP $remote_addr;
    }
} 

[myrodgkim]

Installed versions

• devpi-semantic-ui-0.2.2
• devpi-server-5.5.0
• devpi-web-4.0.6

[fschulze]

A few things I noticed:

• 1GB of RAM isn't enough when using devpi-web and its search. The indexing needs at least 2GB peak memory (mostly on startup, but sometimes for search as well, depending on the query). I would go for 4GB to be save.
• Do not use --start and --stop. Run devpi-server as a foreground process with a process manager like systemd, supervisord or some other one of your choice. With --gen-config you can get some example configuration files for several variants.
• On the server you don't need to install devpi-client

Is only /+search affected by this and the rest of the web interface works? Did you try without the theme (though I don't think it does anything wrong judging from me looking at it a few months back).

[myrodgkim]

@fschulze thanks.
I got it working with 4GB memory and theme removed.
I referenced https://github.com/apihackers/docker-devpi/blob/master/Dockerfile for docker image
So, I just installed devpi-semantic-ui without any specific purpose. but if possible, using theme will be better.

I will test with theme, with more memory configurations.
does devpi-semantic-ui is managed by this group? or by other group?

[fschulze]

The theme and the docker image are external, but as I said: The last time I looked the theme was fine. Can't say anything about the Docker image.

[myrodgkim]

If I use 'supervisor' to manage the processes, How can I collect logs from docker container?
(since documentation says not to use '--start')
I mean with AWS Fargate, I collect stdout/stderr to cloudwatch, but if I use supervisor, there is no output from docker.

Sorry, if it's not appropriate. my best guess is to use shell script to tail-ing log files.

[myrodgkim]

It seems https://github.com/apihackers/ is the author of devpi-semantic-ui module. I will contact them if required.

[fschulze]

I have no idea about Docker, maybe you can even run devpi-server directly as a foreground process as the last command instead of the log tailing and waiting for pid in entrypoint.sh

[fschulze]

Have you been able to solve this?

[myrodgkim]

I removed semantic-ui, and it works.
So, this ticket may be closed.

[fschulze]

Not sure how semantic-ui can causes this, but maybe you should report it to them if you haven't already.