All the APIs showing Offline while working on docker containers

[Urgen-Dorjee]

I have been working on this project for a while and started to deploy on docker containers, but having some issue while doing so, I hope you can give me some ideas about the problem. Docker compose build fine but does not render the pages as it should be. Following are my docker files setup :-

Formerly i have tried to apply docker file to all the projects but after having this problem I tried to start from a project rather than whole therefore, I have tried to recreate/reproduce the problem with a customer management project:-

Docker file CustomerManagement API

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /src
COPY ["src/CustomerManagement/CustomerAPI/CustomerAPI.csproj", "src/CustomerManagement/CustomerAPI/"]

COPY . .
WORKDIR "/src/src/CustomerManagement/CustomerAPI"
RUN dotnet build "CustomerAPI.csproj" -c Release -o /app
RUN dotnet restore -s https://api.nuget.org/v3/index.json -s https://www.myget.org/F/autoweb/api/v3/index.json


FROM build AS publish
RUN dotnet publish "CustomerAPI.csproj" -c Release -o /app

HEALTHCHECK --interval= --timeout= --retries=1 CMD curl --silient --fail http://localhost:5000/hc || exit 1

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CustomerAPI.dll"]

Launch settings.json

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "http://localhost:5000",
      "sslPort": 0
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "index.html",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "CustomerAPI": {
      "commandName": "Project",
      "launchBrowser": true,
      "launchUrl": "index.html",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "applicationUrl": "http://localhost:5000"
    }
  }
}

Docker file Web:-

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /src
COPY ["src/WebManagement/Web/Web.csproj", "src/WebManagement/Web/"]

COPY . .
WORKDIR "/src/src/WebManagement/Web"
RUN dotnet build "Web.csproj" -c Release -o /app
RUN dotnet restore -s https://api.nuget.org/v3/index.json -s https://www.myget.org/F/autoweb/api/v3/index.json


FROM build AS publish
RUN dotnet publish "Web.csproj" -c Release -o /app

HEALTHCHECK --interval=30s --timeout=3s --retries=1 CMD curl --silent --fail http://localhost:7000/hc || exit 1


FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Web.dll"]

Launch settings.json

{
  "iisSettings": {
    "windowsAuthentication": false,
    "anonymousAuthentication": true,
    "iisExpress": {
      "applicationUrl": "http://localhost:7000",
      "sslPort": 0
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "http://localhost:7000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "Web": {
      "commandName": "Project",
      "launchBrowser": true,
      "launchUrl": "http://localhost:7000",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "applicationUrl": "http://localhost:7000"
    }
  }
}

Docker compose.yml file

version: '3.4'

services:

  logserver:
    image: datalust/seq:latest

  sqlserver:
    image: mcr.microsoft.com/mssql/server:2017-latest

  rabbitmq:
    image: rabbitmq:3-management-alpine

  customerapi:
    image: ${DOCKER_REGISTRY-}customerapi
    build:
      context: .
      dockerfile: src/CustomerManagement/CustomerAPI/Dockerfile
    depends_on:
      - sqlserver
      - rabbitmq

  web:
    image: ${DOCKER_REGISTRY-}web
    build:
      context: .
      dockerfile: src/WebManagement/Web/Dockerfile
    depends_on:
      - customerapi

docker-compose.override.yml file

version: '3.4'

services:
  logserver:
    container_name: logserver
    ports:
      - "5341:80"
    environment:
      - ACCEPT_EULA
      
  sqlserver:
    container_name: sqlserver
    ports:
      - "5433:1433"
    volumes:
      - sqlserverdata:/var/opt/mssql
    environment:
      - ACCEPT_EULA=Y
      - MSSQL_PID=Developer
      - SA_PASSWORD=Pass@word

  rabbitmq:
    container_name: rabbitmq
    ports:
      - "15672:15672"
      - "5672:5672"
    volumes:
      - rabbitmqdata:/var/lib/rabbitmq    
    environment:
      - RABBITMQ_DEFAULT_USER=rabbitmquser
      - RABBITMQ_DEFAULT_PASS=Pass@word

  customerapi:
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://0.0.0.0:80
    ports:
      - "5000:80"

  web:
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://0.0.0.0:80
    ports:
      - "7000:80"

volumes:
  sqlserverdata:
    external: true
  rabbitmqdata:
    external: true

When Refit (RestClient) try to fetch a record from the customerapi (GetCustomers) endpoint, where problem started to happen. I don't know how to step into other projects while debugging from docker compose. I have set a break point in couple of places but it won't hit the customerapi controller action method, since it is a micro-service being called by web front end. I have extracted following errors from a long lines of stack traces if you can help:-

[16:02:33 INF] Start processing HTTP request GET http://localhost:5000/api/customers
[16:02:33 INF] Sending HTTP request GET http://localhost:5000/api/customers
"GET CustomerManagement/Index","ai.location.ip":"172.20.0.1","ai.internal.sdkVersion":"rdddsc:2.10.0-32157","ai.internal.nodeName":"ec1567d9102f.(none)"},"data":{"baseType":"RemoteDependencyData","baseData":{"ver":2,"name":"GET /api/customers","id":"|1a08fcc1445823449b91fe1f785d0f1e.da75cb80_1.","data":"http://localhost:5000/api/customers","duration":"00:00:01.5894365","resultCode":"Faulted","success":false,"type":"Http","target":"localhost:5000","properties":{"DeveloperMode":"true","Error":"Cannot assign requested address","AspNetCoreEnvironment":"Production","_MS.ProcessedByMetricExtractors":"(Name:'Dependencies', Ver:'1.1')"}}}}

[11:16:04 INF] Start processing HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 INF] Sending HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 WRN] Error occured during request-execution. Polly will retry. Exception: Cannot assign requested address
web_1          | [11:16:04 INF] Start processing HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 INF] Sending HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 WRN] Error occured during request-execution. Polly will retry. Exception: Cannot assign requested address
web_1          | [11:16:04 INF] Start processing HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 INF] Sending HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 WRN] Error occured during request-execution. Polly will retry. Exception: Cannot assign requested address
web_1          | [11:16:04 INF] Start processing HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 INF] Sending HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 WRN] Error occured during request-execution. Polly will retry. Exception: Cannot assign requested address
web_1          | [11:16:04 INF] Start processing HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 INF] Sending HTTP request GET http://localhost:5000/api/customers
web_1          | [11:16:04 WRN] Error occured during request-execution. Polly will retry. Exception: Cannot assign requested address

I have tried to run your project but it also give me problem same at WorkshopmanagementAPI but it does display customer and vehicle api. I don't understand where is the problem. I have spent quite a bit time to do a research on it, but could not find any answer therefore i have posted for your help. Hope you won't mind it.

Thanks

[EdwinVW]

First, let me ask you why you have created all new docker and docker-compose files? The files I provide with the project work (as described on the Wiki).

When I look at the files and the logging, my initial guess is that it has something to do with the host-names and ports used:

• I see in the logging that localhost is used to connect to the API. This is not correct for the Production environment. The name of the container should be used.
• I see that you used different container-names. You use customerapi whereas in the code customermanagementapi is used.
• I see that you use port 80 (inside the container) and 5000 (on the host) for exposing the customermanagement API. But the webapp is trying to connect to port 5000 for reaching the API. This is not going to work because on docker networking level the API is running on port 80.

I would suggest to try to get it up & running with the provided docker and docker-compose files. Also check whether the API is running correctly. You can do this when you expose the API to localhost (also described on the Wiki page)? First you will need to check whether the API is running correctly.

[Urgen-Dorjee]

Thank you very much that you have take time to look into the issue. Actually i tried to learn by setting up myself to recreate the same project step by step to get into fine grained of the project so I that I can get detailed insight, I very much like your project therefore I have tried myself to reassemble the wonderful garage that you have constructed, Everything is the same besides some name changes like (customermanagementapi to customerapi). I created dockerfile and compose with the help of visual studio 2019 (Container Orchestrator Support). However i have tried to make sure all the configurations and settings are same so I can make it up and running. Initially I have exposed port 5000/tcp as per your settings, but when I open the web site and tried to browse it won't open therefore i have checked/inspected docker container image build configuration :-

{
        "Id": "sha256:ba3c0adc9130943557bf3d7da4c43f60c1b13d49f1c06e1b93ececd1f8e5172a",
        "RepoTags": [
            "web:latest"
        ],
        "RepoDigests": [],
        "Parent": "sha256:53dff1663cf9b326899cef019a1898e2732cad380b22faadf5f602c7d1c9226e",
        "Comment": "",
        "Created": "2019-08-26T11:29:48.865441Z",
        "Container": "fb46531c87246a80ffde2de338678d3ba1005dc8ff3102d592c2cfe5f6ee7b2d",
        "ContainerConfig": {
            "Hostname": "fb46531c8724",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true",
                "ASPNETCORE_VERSION=2.2.6"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "ENTRYPOINT [\"dotnet\" \"CustomerAPI.dll\"]"
            ],
}

No matter what port you exposed but image container ENV key-value pair always set to 80
Here the dockerfile. I have gone through you wiki page and look into docker docs to get the idea, but unfortunately i could not get it. However i still try to do some research myself as i am pretty new to docker world.

FROM mcr.microsoft.com/dotnet/core/aspnet:2.2 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /src
COPY ["CustomerManagement/CustomerAPI/CustomerAPI.csproj", "CustomerManagement/CustomerAPI/"]

COPY . .
WORKDIR /src/CustomerManagement/CustomerAPI
RUN dotnet build "CustomerAPI.csproj" -c Release -o /app
RUN dotnet restore -s https://api.nuget.org/v3/index.json -s https://www.myget.org/F/autoweb/api/v3/index.json


FROM build AS publish
RUN dotnet publish "CustomerAPI.csproj" -c Release -o /app

EXPOSE 5000/tcp
ENV ASPNETCORE_URLS=http://*:5000
HEALTHCHECK --interval= --timeout= --retries=1 CMD curl --silent --fail http://localhost:5000/hc || exit 1

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CustomerAPI.dll"]

I really don't understand how and where did this value come from: "ASPNETCORE_URLS=http://+:80". I have compared with your container image configurations and it is not the same. Your configuration file has additional line

"ExposedPorts": {"7000/tcp": {}}, and "ASPNETCORE_URLS=http://*:7000", as expected

I really don't know if this is the reason for some other but i can see the mismatch.

Your docker container image configurations:-

 {
        "Id": "sha256:e9c4c407ca8a1535ddea819fbca3af351034e7f88ebbb40e5d338cda9a55c7ca",
        "RepoTags": [
            "pitstop/webapp:latest"
        ],
        "RepoDigests": [],
        "Parent": "sha256:d6e540925b7d60c638b07d8f333d35ca8862f99e30f7d30b61c3dfb5b594cf05",
        "Comment": "",
        "Created": "2019-08-26T12:30:02.7671849Z",
        "Container": "daed11c0ae2bf28dbb22e6322ec3498f6a2ec6dd419c3c878c7612fa3822fece",
        "ContainerConfig": {
            "Hostname": "daed11c0ae2b",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "7000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ASPNETCORE_URLS=http://*:7000",
                "DOTNET_RUNNING_IN_CONTAINER=true",
                "ASPNETCORE_VERSION=2.2.6"
            ],
}

Therefore it does not seem to request at http://customerapi:5000/api/customers rather then to http://localhost:5000/api/customer

Since you asked where customer api was working or not, therefore I have checked when container are running and I was able to browse _http://localhost:5000/swagger/index.html _ I can add new customer to the sqlserver container database, and even http://localhost:5000/hc says Healthy

As you have directed i cloned the project to make up and running however workshopmanagementapi does not open rather getting following error:-

[12:43:04 INF] Start processing HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:04 INF] Sending HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:06 WRN] Error occured during request-execution. Polly will retry. Exception: No such device or address
webapp                            | [12:43:06 INF] Start processing HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:06 INF] Sending HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:07 WRN] Error occured during request-execution. Polly will retry. Exception: No such device or address
webapp                            | [12:43:07 INF] Start processing HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:07 INF] Sending HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:09 WRN] Error occured during request-execution. Polly will retry. Exception: No such device or address
webapp                            | [12:43:09 INF] Start processing HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26
webapp                            | [12:43:09 INF] Sending HTTP request GET http://workshopmanagementapi:5200/api/workshopplanning/2019-08-26

[Urgen-Dorjee]

Wow! I finally got it resolved, you know what this is utter painful. I was looking like a hell to go onto check everywhere, I finally got it to work. I know that docker-compose file has order of precedence to startup dependency services but docker file, horrible. I have compared may be hundred times with your docker file and other to make my configurations put in place where is it required but you know it is not the docker file wanted. When I posted for help no one has notice that, until last guy has mentioned about the order therefore I again check and reset the order like below and it works like charm:-

FROM base AS final
WORKDIR /app
COPY --from=publish /app .

EXPOSE 7000/tcp
ENV ASPNETCORE_URLS=http://*:7000

ENTRYPOINT ["dotnet", "Web.dll"]

I was exposing port before publishing the app. But i have seen some of docker config files has port expose at the top just after FROM therefore I thought there might not have any order of precedence, but I was wrong!

BTW I got it work now with one service customerapi and now I will work on other services and I believe I won't face this problem again and last but not the least can you check your WorkshopmanagementAPI some reason it gives me different error. I don't know as of now but I will try myself as well. Thanks for your help and guidance..

Cheers!

[EdwinVW]

Great to see that you found the problem.

Regarding the WorkshopManagementAPI: when you retrieve a workshopplanning for a certain date and there is no existing WorkshopPlanning yet for that date, the API will return an HTTP status-code 404 - Not Found (see the WorkshopPlanningController.GetByDate method in the WorkshopManagementAPI project). Could this be the error you see in the logging? In the web-app, you will not see this error but just an empty overview-screen without maintenancejobs. When you add a new MaintenanceJob, a WorkshopPlanning is automatically created and the API will not return a 404 anymore.

Can you check whether this is the problem with the WorkshopManagementAPI?

[Urgen-Dorjee]

I found the problem and I have been tried to fix it, actually workshopmanagementapi tries to connect to the sqlserver database, however sqlserver database to become ready state take some times, therefore workshopmanagementapi throw an exception. Database connection resiliency does not seem working in this condition.

Stack traces:-

workshopmanagementapi_1       | Application startup exception: System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
workshopmanagementapi_1       |    at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)

workshopmanagementapi_1       |    at System.Data.SqlClient.SqlConnection.Open()
workshopmanagementapi_1       |    at eAutoWeb.WorkShopManagementAPI.Repositories.SqlServerWorkshopPlanningRepository.EnsureDatabase() in /src/WorkShopManagement/WorkShopManagementAPI/Repositories/SqlServerWorkshopPlanningRepository.cs:line 152
workshopmanagementapi_1       | --- End of stack trace from previous location where exception was thrown ---

  public void EnsureDatabase()
        {
            // init db
            using (SqlConnection conn = new SqlConnection(_connectionString.Replace("WorkshopManagementEventStore", "master")))
            {
                conn.Open();

                // create database
                string sql = "if DB_ID('WorkshopManagementEventStore') IS NULL CREATE DATABASE WorkshopManagementEventStore;";

                Policy
                    .Handle<Exception>()
                    .WaitAndRetry(10, r => TimeSpan.FromSeconds(5), (ex, ts) =>
                    { Console.WriteLine("Error connecting to WorkshopManagementEventStore. Retrying in 5 sec."); })
                    .Execute(() => conn.Execute(sql));

 **remove lines for brevity**
}

Database connection resiliency policy seems not working as I have checked the stack traces completely it does not retry even once.

When you run docker-compose first time to start the containers (you can the exception being thrown there)
Now you open another command prompt and try to run >> docker-compose up again this time it would connect to the database and everything working fine because sqlserver database has already started therefore it does not face any issue with connectivity.

I really don't understand how come workshopmanagementapi is being started first then the customermanagementapi and vehiclemanagementapi even though this service has declared almost end of the all the services but still gets fired-up before these services.

Thanks,

[EdwinVW]

I've looked at the code in the EnsureDatabase method and found the issue. It is the call to conn.Open() that is the culprit. This will throw an exception when the SQL Server instance is not yet up. To fix this, I've wrapped all open calls with a retry (812ad30).

[Urgen-Dorjee]

We are good now!!

Thanks