Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'docker-compose up' cannot startup #201

Closed
snecker opened this issue May 25, 2017 · 17 comments
Closed

'docker-compose up' cannot startup #201

snecker opened this issue May 25, 2017 · 17 comments
Labels
bug Something isn't working and you are sure it's a bug!
Milestone
3.1-2017

Comments

@snecker
Copy link

snecker commented May 25, 2017

console print that
image
@wu-sheng
Copy link
Member

@ascrutae Es image didn't start up?

@snecker
Copy link
Author

snecker commented May 25, 2017

yes,i found that es fail to start ,the error is below
image

@snecker
Copy link
Author

snecker commented May 25, 2017

es-server_1 | [2017-05-25T07:04:36,982][INFO ][o.e.n.Node ] [TestNode] initializing ...
es-server_1 | [2017-05-25T07:04:37,120][INFO ][o.e.e.NodeEnvironment ] [TestNode] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [84gb], net total_space [98.3gb], spins? [possibly], types [ext4]
es-server_1 | [2017-05-25T07:04:37,121][INFO ][o.e.e.NodeEnvironment ] [TestNode] heap size [1.9gb], compressed ordinary object pointers [true]
es-server_1 | [2017-05-25T07:04:37,123][INFO ][o.e.n.Node ] [TestNode] node name [TestNode], node ID [mplmo4GcS-mA0emZXcS2YQ]
es-server_1 | [2017-05-25T07:04:37,123][INFO ][o.e.n.Node ] [TestNode] version[5.3.2], pid[1], build[3068195/2017-04-24T16:15:59.481Z], OS[Linux/3.13.0-86-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
es-server_1 | [2017-05-25T07:04:38,397][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [aggs-matrix-stats]
es-server_1 | [2017-05-25T07:04:38,397][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [ingest-common]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [lang-expression]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [lang-groovy]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [lang-mustache]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [lang-painless]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [percolator]
es-server_1 | [2017-05-25T07:04:38,398][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [reindex]
es-server_1 | [2017-05-25T07:04:38,399][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [transport-netty3]
es-server_1 | [2017-05-25T07:04:38,399][INFO ][o.e.p.PluginsService ] [TestNode] loaded module [transport-netty4]
es-server_1 | [2017-05-25T07:04:38,400][INFO ][o.e.p.PluginsService ] [TestNode] no plugins loaded
es-server_1 | [2017-05-25T07:04:41,282][INFO ][o.e.n.Node ] [TestNode] initialized
es-server_1 | [2017-05-25T07:04:41,282][INFO ][o.e.n.Node ] [TestNode] starting ...
es-server_1 | [2017-05-25T07:04:41,380][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: fd:b4:8c:59:bd:e3:5e:a0
es-server_1 | [2017-05-25T07:04:41,448][INFO ][o.e.t.TransportService ] [TestNode] publish_address {192.168.16.2:9300}, bound_addresses {0.0.0.0:9300}
es-server_1 | [2017-05-25T07:04:41,457][INFO ][o.e.b.BootstrapChecks ] [TestNode] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
es-server_1 | ERROR: bootstrap checks failed
es-server_1 | max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
es-server_1 | [2017-05-25T07:04:41,463][INFO ][o.e.n.Node ] [TestNode] stopping ...
es-server_1 | [2017-05-25T07:04:41,500][INFO ][o.e.n.Node ] [TestNode] stopped
es-server_1 | [2017-05-25T07:04:41,500][INFO ][o.e.n.Node ] [TestNode] closing ...
es-server_1 | [2017-05-25T07:04:41,513][INFO ][o.e.n.Node ] [TestNode] closed
skywalking-collector_1 | replace CLUSTER_CURRENT_HOST_NAME with 127.0.0.1
skywalking-collector_1 | replace CLUSTER_CURRENT_PORT with 1000
skywalking-collector_1 | replace CLUSTER_CURRENT_ROLES with WorkersListener
skywalking-collector_1 | replace CLUSTER_SEED_NODES with 127.0.0.1:1000,127.0.0.1:1001
skywalking-collector_1 | replace ES_CLUSTER_NAME with CollectorDBCluster
skywalking3012017_es-server_1 exited with code 78
skywalking-collector_1 | replcae ES_CLUSTER_NODES with es-server:9300
skywalking-collector_1 | replace ES_INDEX_SHARDS_NUMBER with 2
skywalking-collector_1 | replace ES_INDEX_REPLICAS_NUMBER with 0
skywalking-collector_1 | replace HTTP_HOST_NAME with 0.0.0.0
skywalking-collector_1 | replace HTTP_PORT with 12800
skywalking-collector_1 | replace HTTP_CONTEXT_PATH with /
skywalking-collector_1 | sed: -e expression #1, char 24: unknown option to s' skywalking-collector_1 | Starting collector.... skywalking-webui_1 | collector services : skywalking-collector:7001 skywalking-webui_1 | clear config.properties skywalking-webui_1 | /docker-entrypoint.sh: line 7: /usr/local/skywalking-web/config/config/config.properties: No such file or directory skywalking-webui_1 | /docker-entrypoint.sh: line 11: /usr/local/skywalking-web/config/config/config.properties: No such file or directory skywalking-webui_1 | config.properties as following skywalking-webui_1 | cat: /usr/local/skywalking-web/config/config/config.properties: No such file or directory skywalking-webui_1 | Starting web service.... skywalking-webui_1 | skywalking-webui_1 | . ____ _ __ _ _ skywalking-webui_1 | /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ skywalking-webui_1 | ( ( )\___ | '_ | '_| | '_ \/ _ | \ \ \
skywalking-webui_1 | \/ )| |)| | | | | || (| | ) ) ) )
skywalking-webui_1 | ' || .__|| ||| |_, | / / / /
skywalking-webui_1 | =========||==============|/=///_/
skywalking-webui_1 | :: Spring Boot :: (v1.5.2.RELEASE)
skywalking-webui_1 |
skywalking-collector_1 | Exception in thread "main" java.net.UnknownHostException: es-server: Name or service not known
skywalking-collector_1 | at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
skywalking-collector_1 | at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
skywalking-collector_1 | at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
skywalking-collector_1 | at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
skywalking-collector_1 | at java.net.InetAddress.getAllByName(InetAddress.java:1192)
skywalking-collector_1 | at java.net.InetAddress.getAllByName(InetAddress.java:1126)
skywalking-collector_1 | at java.net.InetAddress.getByName(InetAddress.java:1076)
skywalking-collector_1 | at com.a.eye.skywalking.collector.worker.storage.EsClient.boot(EsClient.java:42)
skywalking-collector_1 | at com.a.eye.skywalking.collector.worker.CollectorBootStartUp.main(CollectorBootStartUp.java:22)

@ascrutae
Copy link
Member

This issue caused by that Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low.

You may execute the following command on docker host.

sysctl -w vm.max_map_count=262144

@snecker
Copy link
Author

snecker commented May 27, 2017

i executed that command and the es-server started correctly,but when I access my $IP:8080 in browser i found the console logs another error
Connect to 127.0.0.1:12800 [/127.0.0.1] failed: Connection refused
image

@ascrutae
Copy link
Member

This issue caused by the collector server configuration of webui not changed. I rebuild the image. :-)

@snecker
Copy link
Author

snecker commented May 27, 2017

it works! thx

@wu-sheng
Copy link
Member

@snecker Thanks for your feedback. Closing this issue.

@wu-sheng wu-sheng added this to the 3.1-2017 milestone May 27, 2017
@wu-sheng wu-sheng added the bug Something isn't working and you are sure it's a bug! label May 27, 2017
@aliluyahoho
Copy link

same error with 3.1-2017.
command doesn't working.
sysctl -w vm.max_map_count=262144

@ascrutae
Copy link
Member

Can you post you error message?

@aliluyahoho
Copy link 

skywalking-collector_1  | [2017-07-14 03:06:20:226] [INFO] - org.skywalking.apm.collector.worker.storage.EsClient.parseClusterNodes(EsClient.java:62) - es NODES: es-server:9300
skywalking-collector_1  | Exception in thread "main" java.net.UnknownHostException: es-server: Name or service not known
skywalking-collector_1  |       at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
skywalking-collector_1  |       at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
skywalking-collector_1  |       at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
skywalking-collector_1  |       at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
skywalking-collector_1  |       at java.net.InetAddress.getAllByName(InetAddress.java:1192)
skywalking-collector_1  |       at java.net.InetAddress.getAllByName(InetAddress.java:1126)
skywalking-collector_1  |       at java.net.InetAddress.getByName(InetAddress.java:1076)
skywalking-collector_1  |       at org.skywalking.apm.collector.worker.storage.EsClient.boot(EsClient.java:42)
skywalking-collector_1  |       at org.skywalking.apm.collector.worker.CollectorBootStartUp.main(CollectorBootStartUp.java:22)

@ascrutae ascrutae reopened this Jul 14, 2017
@ascrutae
Copy link
Member

you may be add the following configuration under the es-server node.

 environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

The configuration file should be like this after you add the configuration

es-server:
    image: elasticsearch:5.3
    command: "-Enode.name=TestNode -Enetwork.host=0.0.0.0 -Ehttp.cors.enabled=true -Ehttp.cors.allow-origin=* -Ethread_pool.bulk.queue_size=1000 -Ecluster.name=CollectorDBCluster"
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    expose:
      - "9200"
      - "9300"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9200"]
      interval: 30s
      timeout: 10s
      retries: 5

@aliluyahoho
Copy link

aliluyahoho commented Jul 14, 2017 •
edited
Loading

feel embarrassed.
my cloud machine has 992M mem total , 220M free.
How much mem should be needed at least , to support skywalking to work .

After setting ES_JAVA_OPTS=-Xms150m -Xmx150m, skywalking31_es-server_1 exited with code 137.

@wu-sheng
Copy link
Member

wu-sheng commented Jul 14, 2017 •
edited
Loading

@aliluyahoho The memory requirement isn't up to us. We use the ElasticSearch as persistence service, and this section of docker-compose is for starting it. So it is up to ElasticSearch.

220M seems too less.... and if you only have 220M, how can you be sure the elasticsearch won't slow everything down?

@wu-sheng
Copy link
Member

If you just want to run test or demonstration, the only option I known is waiting for v3.2. In the next version, the PM ( @pengys5 ) intends to provide a standalone mode, which only use H2-in-memory-mode to run our collector, instead of ElasticSearch for easier usage and less memory requirement.

But if you requirement is running in Product env, either docker-compose or standalone mode are NOT a good choice. I highly recommend to use ElasticSearch Cluster.

Hope these help. Let me know, if you have any more question.

BTW, can you help to Start and Fork our project for supporting us? Thanks anyway.

@aliluyahoho
Copy link

@wu-sheng Thx for your question.
To clear up my confusion, I run collector and web on my mac book. They are work properly. Mem condition show as following image.
image

I will get my cloud machine configuration upgrade , then try again .

This time is my personal trial. If it makes our team's requirement, we maybe running in product env.

thx a lot.

@wu-sheng
Copy link
Member

@aliluyahoho Wait for any feedback. Hope 3.2 will help more people, since we added new features about JVM and service, and improved performance.

Closing this. Open this or new, if you have something to discuss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working and you are sure it's a bug!
Projects
None yet
Milestone
3.1-2017
Development

No branches or pull requests
4 participants
@snecker @wu-sheng @aliluyahoho @ascrutae