Problems with mongod

[perfaram]

When launching API-Umbrella, I get :

Failed to start processes:
  distributed-rate-limits-sync (STARTING - /opt/api-umbrella/var/log/supervisor/distributed-rate-limits-sync.log)
  log-processor (STARTING - /opt/api-umbrella/var/log/supervisor/log-processor.log)
  mongod (FATAL - /opt/api-umbrella/var/log/supervisor/mongod.log)

Full output : https://gist.github.com/perfaram/b0da621640fbff0dd52c
Too bad, hmmm ?
I run Debian (Tried CentOS 6.5, 7, Ubuntu 12.04 and 14.04, no luck with any of them, always a bug, real sadness)

If you need anything more, then ask, I'll gladly provide it.

[perfaram]

Anyway, does anybody out there knows a working API manager ? Dont mistake me, I do believe in API-Umbrella, but at this time I just can't even test it with my projects, not to speak about putting in prod'.
I do believe, however, that #86 could make things better (API-Umbrella's architecture is really complicated, and I can only imagine the pain it is to make all that work together.)

[GUI]

I'm very sorry for the trouble. As noted in #89, I'll try to get v0.7 out this weekend which does fix a variety of startup issues. However, if you've been running into this across the board in all those OSes, there might be something else at play (we actively run the packages on several CentOS 6.5 production boxes, and part of our packaging process is in fact verifying that the package installs and runs successfully on each platform).

Would you mind sharing the contents of /opt/api-umbrella/var/log/supervisor/mongod.log and /opt/api-umbrella/var/log/supervisor/supervisord_forever.log? That might help us pinpoint what's going on here.

In terms of other options, Kin Lane has a good recent list of open source API management options. There are also plenty of services that can provide this. All of the offerings obviously vary depending on your needs.

But yes, one of the big goals of #86 would be to simplify the stack, which in turn would hopefully make things easier to run. That being said, we are definitely running this in production now, but the package based installers are new, so we'd greatly appreciate any help you can provide by supplying logs so we can get to the bottom of these issues.

[perfaram]

@GUI : First and foremost, thanks for your answer ! I also came across Lane's API manager list, and it was very instructive to me - I even bookmarked it, in fact. The main goal I have is NOT to use anything external - everything must be under control. I plan to use ApiAxle - but I'm not going to leave our favourite umbrella ! I'll just wait a bit to use it in production.

About the logs :

• supervisord : https://gist.github.com/perfaram/8c54b8874f51f0043929
• mongod :

2015-01-30T15:29:23.776+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:28.833+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:31.687+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:36.410+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:48.578+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:50.304+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:53.110+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.
2015-01-30T15:29:56.841+0000 SEVERE: Failed global initialization: BadValue Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.

• log processor :

Error watching for file: /root/config/runtime.json { [Error: watch EACCES] code: 'EACCES', errno: 'EACCES', syscall: 'watch' }
Error watching for file: /root/config/runtime.json { [Error: watch EACCES] code: 'EACCES', errno: 'EACCES', syscall: 'watch' }
Error watching for file: /root/config/runtime.json { [Error: watch EACCES] code: 'EACCES', errno: 'EACCES', syscall: 'watch' }
Error watching for file: /root/config/runtime.json { [Error: watch EACCES] code: 'EACCES', errno: 'EACCES', syscall: 'watch' }
[Fri Jan 30 2015 15:35:40 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:35:40 GMT+0000 (UTC)] ERROR Log processor worker connections error:  [Error: failed to connect to [127.0.0.1:14001]]

• distributed rate limits :

[Fri Jan 30 2015 15:29:38 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:29:38 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:30:00 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:30:00 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:30:12 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:30:12 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:35:03 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:35:03 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:35:14 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:35:14 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:35:30 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:35:30 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]
[Fri Jan 30 2015 15:35:40 GMT+0000 (UTC)] ERROR Mongo event: disconnected undefined
[Fri Jan 30 2015 15:35:40 GMT+0000 (UTC)] ERROR Distributed rate limits sync connections error:  [Error: failed to connect to [127.0.0.1:14001]]

[perfaram]

The thing is, user locales are set.
Thanks again !
PS : Looking forward for the v0.7 😌

[GUI]

Oh, very interesting, thanks for those logs. This might be something relatively straight forward.. If you get a chance, can you run one more quick test for me? Can you run these two commands on your system and show me the output of each command?

$ env | grep -E "(LC|LANG)"
$ sudo env | grep -E "(LC|LANG)"

Thanks!

And yeah, I definitely encourage you to check out the other API management options. There's a lot of good, exciting stuff happening in this space these days. And if one of them works better for you, and you're still interested in API Umbrella, we'd of course love to know how you think we can improve.

[perfaram]

The first one :

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE=UTF-8

The second one :

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE=UTF-8

(same thing, no matter whether sudo is used or not)

[GUI]

Hm, that's not quite the simple issue I was hoping for, but thank you, that still helps. Can you tell me anything more about this machine? This is Debian, right? Can you tell me which specific version (I think lsb_release -a)? Is this a fresh installation, or is this on an existing server? If it's a fresh install, can you tell me anything about how you installed it (for example, was it just a minimal, basic install, or did you do anything else)? I'll try to reproduce this locally on a VM. Thanks again.

[GUI]

Oh, and if you wouldn't mind, could you also send me the output of locale -a?

If you feel like messing around, it looks like this is an issue other people have ran into with MongoDB. There are a couple of potential workarounds of running sudo locale-gen to generate the missing locales, or forcing the locale to something else via export LC_ALL=C. But I'll try to get to the bottom of this to ensure our default package installation doesn't have this issue.

Thanks for your patience, and sorry again for the trouble.

[perfaram]

VM Specs : 1GB RAM / 30GB SSD / Debian GNU/Linux 7.6 (wheezy) x64
It was the automated install made by DigitalOcean

root@aircraft:~# locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
C
C.UTF-8
en_GB.utf8
en_US.utf8
POSIX

[GUI]

@perfaram: Thanks. I think there's two different issues you've likely been experiencing here:

1. Locale issues (perhaps specific to older DigitalOcean images?)

   • I found several reports of people experiencing similar locale issues with the default DigitialOcean images from last year. However, on a fresh DigitalOcean image for Debian, I did not experience these issues. But my fresh image also seems to be running a newer version of Debian, so is your DigitalOcean image perhaps a bit older? If so, I think you might be able to fix it by running sudo locale-gen. If you then run locale -a does that get rid of the warning you were seeing before? For reference, here's what I see on a fresh DigitalOcean image, where I'm not experiencing locale issues:

     $ lsb_release -a
     No LSB modules are available.
     Distributor ID: Debian
     Description:    Debian GNU/Linux 7.8 (wheezy)
     Release:    7.8
     Codename:   wheezy
     
     $ locale -a
     C
     C.UTF-8
     en_US.utf8
     POSIX
     
     $ env | grep -E "(LC|LANG)"
     LANG=en_US.UTF-8
     LANGUAGE=en_US:en
     LC_CTYPE=en_US.UTF-8

2. Not enough memory
   • Have all your other attempts with other OSes been on DigitalOcean 1GB images? If so, then some of the startup issues may have been due to running out of memory. I added minimum system recommendations to the download page. I'd recommend at least 1.5GB of RAM. It's technically possible to run it with less RAM, but in that case, you'll need to add some swap space, which DigitialOcean appears to disable by default. So if you want to run it on your 1GB image, you probably can with degraded performance, but you'll definitely need to add swap. But I'd probably recommend using the 2GB DigitalOcean image instead, if possible.
   • For the next API Umbrella release I've added warnings to the startup process to help catch low memory situations and better warn users about it, so the failures will be more obvious.
   • As part of Revisiting, speeding up, and simplifying API Umbrella's architecture #86, I think there's a fair amount we can revisit to reduce memory usage, but for now, I think you'll need about 1.5 GB free if you're running all the components on a single server.

Thanks again for reporting these issues.

[perfaram]

@GUI : Thanks to you ! That memory issue is, sadly, not really unexpected... As I tried to run GitLab on that VM - and then I saw that it requires moar memory.
I think that adding warnings is indeed a good thing.
About locale-gen, I think I ran it something between 10 and 15 times - no change.

[perfaram]

Okay, so appears that making the VM bigger makes everything run smoothly. If only I knew...
Thanks !

[GUI]

As a heads up, the new v0.7 packages have been released that should greatly improve these types of startup issues. So hopefully that will make things smoother in the future for you, but thanks for reporting these issues!

[brylie]

We have had locale issues in the past as well. One thing that has helped on Debian was to run:

$ sudo dpkg-reconfigure locales

This, in conjunction with locale-gen was necessary to specify default locales, etc. Also, this may need to be done before installing API Umbrella.

To give a more specific example where I have had locale issues was when running an automated installer for Webmin that included PostgreSQL on a fresh VM with improper locale configuration. The PostgreSQL packages were not successfully installed and I did not catch the error initially. After some head scratching and searching around, I did the following:

1. locale-gen for each needed locale
2. dpkg-reconfigure locales (on Debian) to set proper locales
3. apt-get remove (PostgreSQL)
4. apt-get install (PostgreSQL)

A similar procedure may work for API Umbrella/Mongo, and/or just start from a fresh VM :-)