Self-hosted Gitlab bot stopped doing anything

[egorovli]

Hi. Thanks for the awesome project. We've been using it for a couple of weeks as a self-hosted solution until the bot suddenly stopped doing anything. I've tried setting it up from scratch but with no luck. I will describe step by step what I've done:

1. Created a bot with @BotFather and got a token

999999999:zzzz9zzzzzzzz_zzz9zzz999zzzzzzzzz9z

2. Created new gitlab OAuth application in Gitlab's admin area:

ID=z9zz9zzz99z9999zzz99zz99z9z9z9999z9zz9zzzzzz99zz99z99zzzz9zz9z99
SECRET=99999999zz9zzzz9999999zz9999z9zzz99z99zzzzzzzz9z99z9zzz9z999zz99

3. Created a docker-compose.yml file:

version: '3'

services:
  mongo:
    image: mongo:3.4.10
    expose:
      - "27017"
    command: --smallfiles --port 27017
    volumes:
      - /data/db:/data/db

  redis:
    restart: always
    command: redis-server --appendonly yes
    image: redis:3.2
    expose:
        - "6379"
    volumes:
        - /data/redis:/data

  integram:
    image: integram/integram:latest
    restart: always
    volumes:
      - /data/integram:/app/.conf
    links:
      - mongo
      - redis
    depends_on:
      - mongo
      - redis
    expose:
      - "7000"
    environment:
      - TZ=UTC
      - INTEGRAM_MONGO_URL=mongodb://mongo:27017/integram
      - INTEGRAM_REDIS_URL=redis:6379
      - INTEGRAM_INSTANCE_MODE=multi-main
      - INTEGRAM_CONFIG_DIR=/app/.conf
      - INTEGRAM_PORT=7000
      - INTEGRAM_BASE_URL=https://bot.example.org

  gitlab:
    image: integram/gitlab
    restart: always
    links:
      - mongo
      - redis
    depends_on:
      - integram
    environment:
      - TZ=UTC
      - INTEGRAM_MONGO_URL=mongodb://mongo:27017/integram
      - INTEGRAM_REDIS_URL=redis:6379
      - INTEGRAM_INSTANCE_MODE=multi-service
      - INTEGRAM_BASE_URL=https://bot.example.org
      - GITLAB_BOT_TOKEN=999999999:zzzz9zzzzzzzz_zzz9zzz999zzzzzzzzz9z
      - GITLAB_OAUTH_ID=z9zz9zzz99z9999zzz99zz99z9z9z9999z9zz9zzzzzz99zz99z99zzzz9zz9z99
      - GITLAB_OAUTH_SECRET=99999999zz9zzzz9999999zz9999z9zzz99z99zzzzzzzz9z99z9zzz9z999zz99

  caddy:
    image: abiosoft/caddy
    depends_on:
      - integram
    links:
      - integram
    environment:
      - CADDYPATH=/etc/caddycerts
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/www/Caddyfile:/etc/Caddyfile
      - /var/www/.caddy:/etc/caddycerts
      - /var/www/logs:/srv/logs

When I start the application, I see a bunch of successful logs:

$ docker-compose logs integram

integram_1         | [GIN] 2018/02/16 - 10:00:47 | 200 |    1.076604ms |      172.18.0.1 |  GET     /gitlab/healthcheck
integram_1         | [GIN] 2018/02/16 - 10:01:17 | 200 |    1.266329ms |      172.18.0.1 |  GET     /gitlab/healthcheck
integram_1         | [GIN] 2018/02/16 - 10:01:47 | 200 |    1.126916ms |      172.18.0.1 |  GET     /gitlab/healthcheck

$ docker-compose logs gitlab

gitlab_1           | time="2018-02-16T07:04:54Z" level=info msg="Integram mode: multi-service"
gitlab_1           | time="2018-02-16T07:04:54Z" level=info msg="MongoDB connected: mongodb://mongo:27017/integram"
gitlab_1           | time="2018-02-16T07:04:57Z" level=info msg="gitlab is performing on behalf of @exampleOrgGitlabBot"
gitlab_1           | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
gitlab_1           |  - using env:	export GIN_MODE=release
gitlab_1           |  - using code:	gin.SetMode(gin.ReleaseMode)
gitlab_1           |
gitlab_1           | [GIN-debug] HEAD   /:param1/:param2/:param3  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] GET    /:param1/:param2/:param3  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] POST   /:param1/:param2/:param3  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] HEAD   /:param1/:param2          --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] GET    /:param1/:param2          --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] POST   /:param1/:param2          --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] HEAD   /:param1                  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] GET    /:param1                  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] POST   /:param1                  --> github.com/integram-org/gitlab/vendor/github.com/requilence/integram.serviceHookHandler (5 handlers)
gitlab_1           | [GIN-debug] Listening and serving HTTP on :7000
gitlab_1           | time="2018-02-16T07:04:59Z" level=info msg="gitlab service: workers pool [1] started"
gitlab_1           | [GIN] 2018/02/16 - 07:05:13 | 500 |    1.167077ms |      172.18.0.1 |  GET     /gitlab/healthcheck
gitlab_1           | [GIN] 2018/02/16 - 07:05:43 | 200 |     616.633µs |      172.18.0.1 |  GET     /gitlab/healthcheck
gitlab_1           | [GIN] 2018/02/16 - 07:06:13 | 200 |     721.849µs |      172.18.0.1 |  GET     /gitlab/healthcheck
gitlab_1           | [GIN] 2018/02/16 - 07:06:43 | 200 |     557.026µs |      172.18.0.1 |  GET     /gitlab/healthcheck

Then I tried the following:

1. Send a direct message to the bot: /start
2. Invite the bot to a group and issue some commands: /start@exampleOrgGitlabBot

But nothing seems to happen.

If I inspect mongo, I can see that the bot is receiving the messages:

$ docker-compose exec mongo mongo integram

MongoDB shell version v3.4.10
connecting to: mongodb://127.0.0.1:27017/integram
MongoDB server version: 3.4.10
[...]

> db.messages.find().pretty()
{
	"_id" : ObjectId("5a86aecd18880afb2c85c7fb"),
	"msgid" : 4,
	"botid" : 414731650,
	"fromid" : 1431227,
	"chatid" : 1431227,
	"date" : ISODate("2018-02-16T10:13:33Z"),
	"text" : "/start"
}
{
	"_id" : ObjectId("5a86af0218880afb2c85c830"),
	"msgid" : 6,
	"botid" : 414731650,
	"fromid" : 1431227,
	"chatid" : -262650351,
	"date" : ISODate("2018-02-16T10:14:26Z")
}
{
	"_id" : ObjectId("5a86af0a18880afb2c85c83a"),
	"msgid" : 7,
	"botid" : 414731650,
	"fromid" : 1431227,
	"chatid" : -262650351,
	"date" : ISODate("2018-02-16T10:14:34Z"),
	"text" : "/start@exampleOrgGitlabBot"
}

> db.chats.find().pretty()
{
	"_id" : NumberLong(1431227),
	"type" : "private",
	"firstname" : "Anton",
	"lastname" : "Egorov",
	"username" : "xxx",
	"createdat" : ISODate("2018-02-16T10:13:33.386Z"),
	"hooks" : [
		{
			"token" : "cJ5GU2Ug2q6",
			"services" : [
				"gitlab"
			]
		}
	]
}
{
	"_id" : NumberLong(-262650351),
	"type" : "group",
	"firstname" : "",
	"title" : "zzz",
	"createdat" : ISODate("2018-02-16T10:14:26.304Z"),
	"hooks" : [
		{
			"token" : "c4PvppgFA5z",
			"services" : [
				"gitlab"
			]
		}
	],
	"membersids" : [
		NumberLong(1431227),
		NumberLong(3343033)
	]
}

Any hint is really much appreciated, thanks!

[requilence]

Hi! This is weird. Looks like outgoing messages from bot failed to sent.

Could you please add the env var INTEGRAM_DEBUG=1 to both integram and gitlab containers? This will make logs more verbose.

[egorovli]

Ok, I know this is weird, but I've added INTEGRAM_DEBUG=1 to both integram and gitlab, and after the restart all unsent messages got sent, like 50 of them at once. I'll e-mail you the logs it that's ok.

[requilence]

@egorovli Looks like it was some problem with Redis queue. BTW feel free to write me on TG @requilence if this will happen again