Skip to content
This repository has been archived by the owner on Sep 2, 2020. It is now read-only.

Cannot render metrics in graphana #130

Closed
valentin-fischer opened this issue Sep 21, 2015 · 5 comments
Closed

Cannot render metrics in graphana #130

valentin-fischer opened this issue Sep 21, 2015 · 5 comments

Comments

@valentin-fischer
Copy link

Hi guys,

I just made a logging setup composed of

  • graphana (2.1.3)
  • graphite-api (latest)
  • cyanite (latest)
  • cassandra (latest)

Unfortunately after spending hours and hours trying to graph some metrics in graphana, it's simple impossible to make it work.
It seems that cyanite is crashing at some point and refusing to communicate with graphite-api after some point in time.

I'm starting cyanite using:
java -cp ./target/cyanite-0.5.1-standalone.jar io.cyanite

INFO [2015-09-21 09:22:21,290] main - io.cyanite installing signal handlers
INFO [2015-09-21 09:22:21,298] main - io.cyanite ready to start system
DEBUG [2015-09-21 09:22:21,352] main - com.datastax.driver.core.Cluster Starting new cluster with contact points [localhost/127.0.0.1:9042]
DEBUG [2015-09-21 09:22:21,377] main - com.datastax.shaded.netty.util.internal.logging.InternalLoggerFactory Using SLF4J as the default logging framework
DEBUG [2015-09-21 09:22:21,390] main - com.datastax.shaded.netty.util.internal.PlatformDependent0 java.nio.Buffer.address: available
DEBUG [2015-09-21 09:22:21,390] main - com.datastax.shaded.netty.util.internal.PlatformDependent0 sun.misc.Unsafe.theUnsafe: available
DEBUG [2015-09-21 09:22:21,390] main - com.datastax.shaded.netty.util.internal.PlatformDependent0 sun.misc.Unsafe.copyMemory: available
DEBUG [2015-09-21 09:22:21,390] main - com.datastax.shaded.netty.util.internal.PlatformDependent0 java.nio.Bits.unaligned: true
DEBUG [2015-09-21 09:22:21,393] main - com.datastax.shaded.netty.util.internal.PlatformDependent Java version: 8
DEBUG [2015-09-21 09:22:21,395] main - com.datastax.shaded.netty.util.internal.PlatformDependent -Dio.netty.noUnsafe: false
DEBUG [2015-09-21 09:22:21,395] main - com.datastax.shaded.netty.util.internal.PlatformDependent sun.misc.Unsafe: available
DEBUG [2015-09-21 09:22:21,395] main - com.datastax.shaded.netty.util.internal.PlatformDependent -Dio.netty.noJavassist: false
DEBUG [2015-09-21 09:22:21,396] main - com.datastax.shaded.netty.util.internal.PlatformDependent Javassist: unavailable
DEBUG [2015-09-21 09:22:21,396] main - com.datastax.shaded.netty.util.internal.PlatformDependent You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes.  Please check the conf
iguration for better performance.
DEBUG [2015-09-21 09:22:21,398] main - com.datastax.shaded.netty.util.internal.PlatformDependent -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
DEBUG [2015-09-21 09:22:21,398] main - com.datastax.shaded.netty.util.internal.PlatformDependent -Dio.netty.bitMode: 64 (sun.arch.data.model)
DEBUG [2015-09-21 09:22:21,398] main - com.datastax.shaded.netty.util.internal.PlatformDependent -Dio.netty.noPreferDirect: false
DEBUG [2015-09-21 09:22:21,410] main - com.datastax.driver.core.SystemProperties com.datastax.driver.FORCE_NIO is undefined, using default value false
INFO [2015-09-21 09:22:21,412] main - com.datastax.driver.core.NettyUtil Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
DEBUG [2015-09-21 09:22:21,417] main - com.datastax.shaded.netty.channel.MultithreadEventLoopGroup -Dio.netty.eventLoopThreads: 2
DEBUG [2015-09-21 09:22:21,427] main - com.datastax.shaded.netty.channel.nio.NioEventLoop -Dio.netty.noKeySetOptimization: false
DEBUG [2015-09-21 09:22:21,428] main - com.datastax.shaded.netty.channel.nio.NioEventLoop -Dio.netty.selectorAutoRebuildThreshold: 512
DEBUG [2015-09-21 09:22:21,451] main - com.datastax.shaded.netty.util.ResourceLeakDetector -Dcom.datastax.shaded.netty.leakDetectionLevel: simple
DEBUG [2015-09-21 09:22:21,692] main - com.datastax.driver.core.SystemProperties com.datastax.driver.DISABLE_COALESCING is undefined, using default value false
DEBUG [2015-09-21 09:22:21,729] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.numHeapArenas: 1
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.numDirectArenas: 1
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.pageSize: 8192
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.maxOrder: 11
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.chunkSize: 16777216
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.tinyCacheSize: 512
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.smallCacheSize: 256
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.normalCacheSize: 64
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.maxCachedBufferCapacity: 32768
DEBUG [2015-09-21 09:22:21,730] main - com.datastax.shaded.netty.buffer.PooledByteBufAllocator -Dio.netty.allocator.cacheTrimInterval: 8192
WARN [2015-09-21 09:22:24,776] main - com.datastax.shaded.netty.util.internal.ThreadLocalRandom Failed to generate a seed from SecureRandom within 3 seconds. Not enough entrophy?
DEBUG [2015-09-21 09:22:24,777] main - com.datastax.shaded.netty.util.internal.ThreadLocalRandom -Dio.netty.initialSeedUniquifier: 0x73c1d6d6c618c119 (took 3003 ms)
DEBUG [2015-09-21 09:22:24,794] main - com.datastax.shaded.netty.buffer.ByteBufUtil -Dio.netty.allocator.type: unpooled
DEBUG [2015-09-21 09:22:24,794] main - com.datastax.shaded.netty.buffer.ByteBufUtil -Dio.netty.threadLocalDirectBufferSize: 65536
DEBUG [2015-09-21 09:22:25,015] cluster1-nio-worker-0 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] Connection opened successfully
DEBUG [2015-09-21 09:22:25,064] cluster1-nio-worker-0 - com.datastax.shaded.netty.util.Recycler -Dio.netty.recycler.maxCapacity.default: 262144
DEBUG [2015-09-21 09:22:25,147] main - com.datastax.driver.core.ControlConnection [Control connection] Refreshing node list and token map
DEBUG [2015-09-21 09:22:25,350] main - com.datastax.driver.core.ControlConnection [Control connection] Refreshing schema
DEBUG [2015-09-21 09:22:25,465] main - com.datastax.driver.core.ControlConnection [Control connection] Refreshing node list and token map
DEBUG [2015-09-21 09:22:25,543] main - com.datastax.driver.core.ControlConnection [Control connection] Successfully connected to localhost/127.0.0.1:9042
INFO [2015-09-21 09:22:25,543] main - com.datastax.driver.core.policies.DCAwareRoundRobinPolicy Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwar
eRoundRobinPolicy constructor)
INFO [2015-09-21 09:22:25,543] main - com.datastax.driver.core.Cluster New Cassandra host localhost/127.0.0.1:9042 added
DEBUG [2015-09-21 09:22:25,566] cluster1-nio-worker-1 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-2, inFlight=0, closed=false] Connection opened successfully
DEBUG [2015-09-21 09:22:25,586] cluster1-nio-worker-1 - com.datastax.driver.core.Session Added connection pool for localhost/127.0.0.1:9042
INFO [2015-09-21 09:22:25,669] main - io.cyanite.engine starting engine
DEBUG [2015-09-21 09:22:25,719] main - io.netty.channel.MultithreadEventLoopGroup -Dio.netty.eventLoopThreads: 2
DEBUG [2015-09-21 09:22:25,744] main - io.netty.channel.nio.NioEventLoop -Dio.netty.noKeySetOptimization: false
DEBUG [2015-09-21 09:22:25,745] main - io.netty.channel.nio.NioEventLoop -Dio.netty.selectorAutoRebuildThreshold: 512
WARN [2015-09-21 09:22:28,763] main - io.netty.util.internal.ThreadLocalRandom Failed to generate a seed from SecureRandom within 3 seconds. Not enough entrophy?
DEBUG [2015-09-21 09:22:28,763] main - io.netty.util.internal.ThreadLocalRandom -Dio.netty.initialSeedUniquifier: 0x87d233603e18c119 (took 3003 ms)
DEBUG [2015-09-21 09:22:28,803] main - io.netty.buffer.ByteBufUtil -Dio.netty.allocator.type: unpooled
DEBUG [2015-09-21 09:22:28,806] main - io.netty.buffer.ByteBufUtil -Dio.netty.threadLocalDirectBufferSize: 65536
DEBUG [2015-09-21 09:22:28,809] main - io.netty.util.NetUtil Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
DEBUG [2015-09-21 09:22:28,810] main - io.netty.util.NetUtil /proc/sys/net/core/somaxconn: 128
DEBUG [2015-09-21 09:22:28,854] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle org.eclipse.jetty.server.Server@781acaa8 added {qtp657244232{STOPPED,8<=0<=50,i=0,q=0},AUTO}
DEBUG [2015-09-21 09:22:28,856] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle org.eclipse.jetty.server.Server@781acaa8 added {org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@3c0faed4,AUTO}
DEBUG [2015-09-21 09:22:28,872] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle HttpConnectionFactory@51eb5e67[HTTP/1.1] added {HttpConfiguration@56952bbe{32768/8192,8192/8192,https://:443,[]},POJO}
DEBUG [2015-09-21 09:22:28,879] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.server.Server@781acaa8,UNMANAGED}
DEBUG [2015-09-21 09:22:28,880] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{null,[]}{0.0.0.0:0} added {qtp657244232{STOPPED,8<=0<=50,i=0,q=0},AUTO}
DEBUG [2015-09-21 09:22:28,881] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@3c0faed4,AUTO}
DEBUG [2015-09-21 09:22:28,881] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.io.ArrayByteBufferPool@f0d7950,POJO}
DEBUG [2015-09-21 09:22:28,883] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{null,[http/1.1]}{0.0.0.0:0} added {HttpConnectionFactory@51eb5e67[HTTP/1.1],AUTO}
DEBUG [2015-09-21 09:22:28,885] claypoole-2-0 - org.eclipse.jetty.server.ServerConnector ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:0} added HttpConnectionFactory@51eb5e67[HTTP/1.1]
DEBUG [2015-09-21 09:22:28,888] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:0} added {org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@6ba0d6e7,M
ANAGED}
DEBUG [2015-09-21 09:22:28,888] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle org.eclipse.jetty.server.Server@781acaa8 added {ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090},AUTO}
DEBUG [2015-09-21 09:22:28,894] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle org.eclipse.jetty.server.handler.HandlerList@63379410[qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a@675
8df33] added {qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a@6758df33,AUTO}
DEBUG [2015-09-21 09:22:28,895] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle org.eclipse.jetty.server.Server@781acaa8 added {org.eclipse.jetty.server.handler.HandlerList@63379410[qbits.jet.server.proxy$org.eclipse.j
etty.server.handler.AbstractHandler$ff19274a@6758df33],AUTO}
DEBUG [2015-09-21 09:22:28,895] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting org.eclipse.jetty.server.Server@781acaa8
INFO [2015-09-21 09:22:28,901] claypoole-2-0 - org.eclipse.jetty.server.Server jetty-9.3.z-SNAPSHOT
DEBUG [2015-09-21 09:22:28,921] claypoole-2-0 - org.eclipse.jetty.server.handler.AbstractHandler starting org.eclipse.jetty.server.Server@781acaa8
DEBUG [2015-09-21 09:22:28,922] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting qtp657244232{STOPPED,8<=0<=50,i=0,q=0}
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12717ms qtp657244232{STARTED,8<=8<=50,i=7,q=0}
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@3c0faed4
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12718ms org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@3c0faed4
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting org.eclipse.jetty.server.handler.HandlerList@63379410[qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19
274a@6758df33]
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.server.handler.AbstractHandler starting org.eclipse.jetty.server.handler.HandlerList@63379410[qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff1927
4a@6758df33]
DEBUG [2015-09-21 09:22:28,924] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a@6758df33
DEBUG [2015-09-21 09:22:28,925] claypoole-2-0 - org.eclipse.jetty.server.handler.AbstractHandler starting qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a@6758df33
DEBUG [2015-09-21 09:22:28,925] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12718ms qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a@6758df33
DEBUG [2015-09-21 09:22:28,925] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12718ms org.eclipse.jetty.server.handler.HandlerList@63379410[qbits.jet.server.proxy$org.eclipse.jetty.server.handler.AbstractHand
ler$ff19274a@6758df33]
DEBUG [2015-09-21 09:22:28,925] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
DEBUG [2015-09-21 09:22:28,928] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090} added {sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:8090],POJO}
DEBUG [2015-09-21 09:22:28,929] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting HttpConnectionFactory@51eb5e67[HTTP/1.1]
DEBUG [2015-09-21 09:22:28,929] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12722ms HttpConnectionFactory@51eb5e67[HTTP/1.1]
DEBUG [2015-09-21 09:22:28,929] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@6ba0d6e7
DEBUG [2015-09-21 09:22:28,932] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle starting org.eclipse.jetty.io.ManagedSelector@3d9ddbbf id=0 keys=-1 selected=-1
DEBUG [2015-09-21 09:22:28,933] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12726ms org.eclipse.jetty.io.ManagedSelector@3d9ddbbf id=0 keys=0 selected=0
DEBUG [2015-09-21 09:22:28,933] claypoole-2-0 - org.eclipse.jetty.util.thread.QueuedThreadPool queue org.eclipse.jetty.io.ManagedSelector@3d9ddbbf id=0 keys=0 selected=0
DEBUG [2015-09-21 09:22:28,934] qtp657244232-34 - org.eclipse.jetty.util.thread.QueuedThreadPool run org.eclipse.jetty.io.ManagedSelector@3d9ddbbf id=0 keys=0 selected=0
DEBUG [2015-09-21 09:22:28,934] qtp657244232-34 - org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume EPR Idle/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@dfde8ed execute
DEBUG [2015-09-21 09:22:28,934] qtp657244232-34 - org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume EPR Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@dfde8ed produce enter
DEBUG [2015-09-21 09:22:28,934] qtp657244232-34 - org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume EPR Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@dfde8ed producing
DEBUG [2015-09-21 09:22:28,934] qtp657244232-34 - org.eclipse.jetty.io.ManagedSelector Selector loop waiting on select
DEBUG [2015-09-21 09:22:28,935] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12728ms org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@6ba0d6e7
DEBUG [2015-09-21 09:22:28,935] claypoole-2-0 - org.eclipse.jetty.util.component.ContainerLifeCycle ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090} added {acceptor-0@14888435,POJO}
DEBUG [2015-09-21 09:22:28,936] claypoole-2-0 - org.eclipse.jetty.util.thread.QueuedThreadPool queue acceptor-0@14888435
DEBUG [2015-09-21 09:22:28,936] qtp657244232-35 - org.eclipse.jetty.util.thread.QueuedThreadPool run acceptor-0@14888435
INFO [2015-09-21 09:22:28,938] claypoole-2-0 - org.eclipse.jetty.server.ServerConnector Started ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
DEBUG [2015-09-21 09:22:28,939] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12732ms ServerConnector@58fd37dd{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
INFO [2015-09-21 09:22:28,939] claypoole-2-0 - org.eclipse.jetty.server.Server Started @12732ms
DEBUG [2015-09-21 09:22:28,940] claypoole-2-0 - org.eclipse.jetty.util.component.AbstractLifeCycle STARTED @12733ms org.eclipse.jetty.server.Server@781acaa8
DEBUG [2015-09-21 09:22:37,659] cluster1-timeouter-0 - com.datastax.driver.core.RequestHandler onTimeout triggered but the response was completed by another thread, cancelling (retryCount = 0, queryState = QueryState(count=0, inProgress=f
alse, cancelled=false), queryStateRef = QueryState(count=0, inProgress=false, cancelled=false))
DEBUG [2015-09-21 09:22:55,473] cluster1-nio-worker-0 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
DEBUG [2015-09-21 09:22:55,477] cluster1-nio-worker-0 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] heartbeat query succeeded
DEBUG [2015-09-21 09:22:55,647] cluster1-nio-worker-1 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
DEBUG [2015-09-21 09:22:55,651] cluster1-nio-worker-1 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-2, inFlight=0, closed=false] heartbeat query succeeded

Inserting values over 2003/nc works .

DEBUG [2015-09-21 09:24:41,058] claypoole-1-0 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827475, :mean 4.0, :max 4.0, :min 4.0, :sum 4328.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:24:46,011] claypoole-1-1 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827480, :mean 4.0, :max 4.0, :min 4.0, :sum 7744.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:24:51,008] claypoole-1-2 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827485, :mean 4.0, :max 4.0, :min 4.0, :sum 8564.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:24:55,488] cluster1-nio-worker-0 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
DEBUG [2015-09-21 09:24:55,491] cluster1-nio-worker-0 - com.datastax.driver.core.Connection Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] heartbeat query succeeded
DEBUG [2015-09-21 09:24:56,007] claypoole-1-3 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827490, :mean 4.0, :max 4.0, :min 4.0, :sum 7696.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:24:58,060] cluster1-timeouter-0 - com.datastax.driver.core.RequestHandler onTimeout triggered but the response was completed by another thread, cancelling (retryCount = 0, queryState = QueryState(count=0, inProgress=false, cancelled=false), queryStateRef = QueryState(count=0, inProgress=false, cancelled=false))
DEBUG [2015-09-21 09:25:01,011] claypoole-1-0 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827495, :mean 4.0, :max 4.0, :min 4.0, :sum 8580.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:25:03,059] cluster1-timeouter-0 - com.datastax.driver.core.RequestHandler onTimeout triggered but the response was completed by another thread, cancelling (retryCount = 0, queryState = QueryState(count=0, inProgress=false, cancelled=false), queryStateRef = QueryState(count=0, inProgress=false, cancelled=false))
DEBUG [2015-09-21 09:25:06,007] claypoole-1-1 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "local.random.diceroll", :time 1442827500, :mean 4.0, :max 4.0, :min 4.0, :sum 8032.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-21 09:25:08,059] cluster1-timeouter-0 - com.datastax.driver.core.RequestHandler onTimeout triggered but the response was completed by another thread, cancelling (retryCount = 0, queryState = QueryState(count=0, inProgress=false, cancelled=false), queryStateRef = QueryState(count=0, inProgress=false, cancelled=false))

The data in cassandra looks good.

cqlsh:metric> SELECT * FROM metric ;

 id                                                                        | time       | point
---------------------------------------------------------------------------+------------+--------------------------------------
 {path: 'local.random.diceroll', resolution: {precision: 5, period: 3600}} | 1442827475 | {max: 4, mean: 4, min: 4, sum: 4328}
 {path: 'local.random.diceroll', resolution: {precision: 5, period: 3600}} | 1442827480 | {max: 4, mean: 4, min: 4, sum: 7744}
 {path: 'local.random.diceroll', resolution: {precision: 5, period: 3600}} | 1442827485 | {max: 4, mean: 4, min: 4, sum: 8564}

(3 rows)

After inserting some values into cyanite/cassandra, graphana is unable to query cyanite anymore and it the following errors start to appear

[pid: 15533|app: 0|req: 456/465] 1.2.3.5 () {40 vars in 686 bytes} [Mon Sep 21 09:28:28 2015] GET /metrics/find?query=* => generated 291 bytes in 22 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

So, my question is, why is the pulling of the metrics failing ?
@ghost
Copy link

ghost commented Sep 28, 2015

Hey @valentinul,

can you provide more information about how you installed your setup? I've setup mine today and already gathering metrics and I'm also able to use them in Grafana via Graphite-API:

SELECT * FROM metric.metric ;
 tenant | period | rollup | path                                                              | time       | data
--------+--------+--------+-------------------------------------------------------------------+------------+------------
        |  60480 |     10 | collectddj-cassandra_unifycollectd.interface-venet0.if_packets.rx | 1443431150 |      [0.1]
        |  60480 |     10 | collectddj-cassandra_unifycollectd.interface-venet0.if_packets.rx | 1443431160 |      [0.2]

Maybe I can help you out somehow. My setup looks as following:

Grafana (2.5.0-pre1, commit: v2.1.2+444-g9da5ef3)
Graphite-API (1.0.2)
Cyanite (0.1)
Cassandra (2.2.1)

screen shot 2015-09-28 at 11 40 05

Cheers,
Szop

@ghost
Copy link

ghost commented Sep 28, 2015

Ok I've managed it to install Cyanite 0.5.1. Please provide your steps so I can help you out and compare with mine.

Cheers,
Szop

@ghost
Copy link

ghost commented Sep 28, 2015

It seem's like I've run into a annoying bug. Graphite-API and Cyanite are playing well until collectd/diamonds sends it's first metrics to port 2003 (carbon on Cyanite):

tail -f /var/log/cyanite/cyanite.log
INFO [2015-09-28 15:04:04,551] main - com.datastax.driver.core.Cluster New Cassandra host /127.0.0.1:9042 added
INFO [2015-09-28 15:04:04,615] main - io.cyanite.engine starting engine
INFO [2015-09-28 15:04:04,707] claypoole-2-0 - org.eclipse.jetty.server.Server jetty-9.3.z-SNAPSHOT
INFO [2015-09-28 15:04:04,724] claypoole-2-0 - org.eclipse.jetty.server.ServerConnector Started ServerConnector@36c01317{HTTP/1.1,[http/1.1]}{10.6.232.1:8080}
INFO [2015-09-28 15:04:04,724] claypoole-2-0 - org.eclipse.jetty.server.Server Started @4192ms
DEBUG [2015-09-28 15:04:17,252] claypoole-1-3 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.df-root.df_complex-reserved", :time 1443445445, :mean 0.0, :max 0.0, :min 0.0, :sum 0.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-28 15:04:17,252] claypoole-1-1 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.interface-lo.if_errors.tx", :time 1443445445, :mean 0.0, :max 0.0, :min 0.0, :sum 0.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-28 15:04:17,253] claypoole-1-0 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.df-root.df_complex-used", :time 1443445445, :mean 1.77610752E9, :max 1.77610752E9, :min 1.77610752E9, :sum 1.77610752E9, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-28 15:04:17,266] claypoole-1-2 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.df-root.df_complex-free", :time 1443445445, :mean 5.191098368E10, :max 5.191098368E10, :min 5.191098368E10, :sum 5.191098368E10, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-28 15:04:17,272] claypoole-1-0 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.interface-lo.if_errors.rx", :time 1443445445, :mean 0.0, :max 0.0, :min 0.0, :sum 0.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}
DEBUG [2015-09-28 15:04:17,278] claypoole-1-3 - io.cyanite.engine in write callback:  #io.cyanite.engine.buckets.MetricSnapshot{:path "dj-cassandra2_unify.interface-venet0.if_packets.rx", :time 1443445445, :mean 0.0, :max 0.0, :min 0.0, :sum 0.0, :resolution #io.cyanite.engine.rule.Resolution{:precision 5, :period 3600}}

tail -f /var/log/apache2/*log
dj-cassandra2.unify:8013 10.6.232.1 - - [28/Sep/2015:15:04:14 +0200] "GET /metrics/find?query=* HTTP/1.1" 200 140 "-" "curl/7.35.0"
dj-cassandra2.unify:8013 10.6.232.1 - - [28/Sep/2015:15:04:15 +0200] "GET /metrics/find?query=* HTTP/1.1" 200 140 "-" "curl/7.35.0"
dj-cassandra2.unify:8013 10.6.232.1 - - [28/Sep/2015:15:04:16 +0200] "GET /metrics/find?query=* HTTP/1.1" 200 140 "-" "curl/7.35.0"
dj-cassandra2.unify:8013 10.6.232.1 - - [28/Sep/2015:15:04:17 +0200] "GET /metrics/find?query=* HTTP/1.1" 500 462 "-" "curl/7.35.0"

curl http://10.6.232.1:8013/metrics/find?query=*
[][][]
!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

After that I've to restart Cyanite :/. It was working without a problem with version 0.1

@ghost
Copy link

ghost commented Sep 29, 2015

I've found out,

my Graphite-API is not "crashing" anymore with commit: 5433d57, but I'm still not able to receive Metrics in Grafana.

Cheers,
Szop

@brutasse brutasse mentioned this issue Sep 30, 2015
Closed
@valentin-fischer
Copy link
Author

Hi guys,

Yes, the issue is present in the latest code but I'm going to try the #14 patch and see if that works for me. Thank you @szop85 and @brutasse for the time and effort invested in this issue :)

All the best,
Valentin

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@valentin-fischer