Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with running simple python example on our remote dev environment on k8s (python directory seems empty) #695

Closed
arminwoworsky opened this issue Nov 6, 2023 · 4 comments
Closed

Issue with running simple python example on our remote dev environment on k8s (python directory seems empty) #695

arminwoworsky opened this issue Nov 6, 2023 · 4 comments

Comments

@arminwoworsky
Copy link

arminwoworsky commented Nov 6, 2023 •
edited
Loading

I am trying to deploy and get a very simple python example (logging message) up and running on our remote dev server. locally with langstream docker run ..., everything runs fine, but it seems that on the remote environment the python code directory is empty. I have no idea what I am missing to get it working. Any hint is highly appreciated.

Thanks Armin

logs from the pod:

20:52:31.936 [main] INFO a.l.runtime.agent.AgentRunnerStarter -- Loading pod configuration from /app-config/config
20:52:32.011 [main] INFO a.l.runtime.agent.AgentRunnerStarter -- Loading code from /app-code-download
20:52:32.011 [main] INFO a.l.runtime.agent.AgentRunnerStarter -- Loading agents from /app/agents
20:52:32.011 [main] INFO a.l.runtime.agent.AgentRunnerStarter -- No persistent state directory
20:52:32.216 [main] INFO a.l.runtime.agent.AgentRunner -- Starting agent langstream-app-4-pipeline-python-sink-1
20:52:32.216 [main] INFO a.l.runtime.agent.AgentRunner -- Code directory /app-code-download
20:52:32.216 [main] INFO a.l.runtime.agent.AgentRunner -- Base persistent state directory null
20:52:32.216 [main] INFO a.l.runtime.agent.AgentRunner -- Looking for java lib in /app-code-download/java/lib
20:52:32.218 [main] INFO a.langstream.impl.nar.NarFileHandler -- langstream.nar.closeClassloaders = true
20:52:32.316 [main] INFO a.l.a.r.t.TopicConnectionsRuntimeRegistry -- Loading code for type pulsar
20:52:32.320 [main] INFO a.langstream.impl.nar.NarFileHandler -- Unpacking NAR file /app/agents/langstream-pulsar-runtime-0.4.2-nar.nar to /tmp/nar4142330129485350395/langstream-pulsar-runtime-0.4.2-nar.nar.dir
20:52:33.844 [main] INFO a.l.a.r.t.TopicConnectionsRuntimeRegistry -- Found the package the type belongs to: langstream-pulsar-runtime-0.4.2-nar.nar
20:52:34.014 [main] INFO a.l.runtime.agent.AgentRunner -- Bootstrapping agent pipeline-python-sink-1 type python-sink
20:52:34.014 [main] INFO a.l.a.runner.code.AgentCodeRegistry -- Loading agent code for type python-sink
20:52:34.017 [main] INFO a.langstream.impl.nar.NarFileHandler -- Unpacking NAR file /app/agents/langstream-agent-grpc-0.4.2-nar.nar to /tmp/nar4142330129485350395/langstream-agent-grpc-0.4.2-nar.nar.dir
20:52:34.555 [main] INFO a.l.a.runner.code.AgentCodeRegistry -- Found the package the agent belongs to: langstream-agent-grpc-0.4.2-nar.nar
20:52:34.922 [main] INFO a.l.runtime.agent.AgentRunner -- Started metrics and agent server on port 8080
20:52:34.924 [main] INFO a.l.runtime.agent.AgentRunner -- The addresses should be ....
20:52:35.027 [main] INFO org.eclipse.jetty.server.Server -- jetty-11.0.15; built: 2023-04-11T18:37:53.775Z; git: 5bc5e562c8d05c5862505aebe5cf83a61bdbcb96; jvm 17.0.8.1+1-Ubuntu-0ubuntu122.04
20:52:35.229 [main] INFO o.e.j.server.handler.ContextHandler -- Started o.e.j.s.ServletContextHandler@7b8233cd{/,null,AVAILABLE}
20:52:35.320 [main] INFO o.e.jetty.server.AbstractConnector -- Started ServerConnector@e54303{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
20:52:35.321 [main] INFO org.eclipse.jetty.server.Server -- Started Server@4983159f{STARTING}[11.0.15,sto=0] @4935ms
20:52:37.212 [main] WARN o.a.p.c.util.netty.DnsResolverUtil -- Cannot get DNS TTL settings from sun.net.InetAddressCachePolicy class
java.lang.IllegalAccessException: class org.apache.pulsar.common.util.netty.DnsResolverUtil cannot access class sun.net.InetAddressCachePolicy (in module java.base) because module java.base does not export sun.net to unnamed module @2c1dc8e
at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
at java.base/java.lang.reflect.Method.invoke(Method.java:560)
at org.apache.pulsar.common.util.netty.DnsResolverUtil.(DnsResolverUtil.java:46)
at org.apache.pulsar.client.impl.ConnectionPool.createAddressResolver(ConnectionPool.java:159)
at org.apache.pulsar.client.impl.ConnectionPool.lambda$new$1(ConnectionPool.java:126)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:126)
at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:95)
at org.apache.pulsar.client.impl.ConnectionPool.(ConnectionPool.java:90)
at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:196)
at org.apache.pulsar.client.impl.PulsarClientImpl.(PulsarClientImpl.java:153)
at org.apache.pulsar.client.impl.ClientBuilderImpl.build(ClientBuilderImpl.java:63)
at ai.langstream.pravega.PulsarClientUtils.buildPulsarClient(PulsarClientUtils.java:61)
at ai.langstream.pravega.runner.PulsarTopicConnectionsRuntimeProvider$PulsarTopicConnectionsRuntime.init(PulsarTopicConnectionsRuntimeProvider.java:106)
at ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader$1.lambda$init$0(TopicConnectionsRuntimeAndLoader.java:74)
at ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader.executeNoExceptionWithContextClassloader(TopicConnectionsRuntimeAndLoader.java:42)
at ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader$1.init(TopicConnectionsRuntimeAndLoader.java:74)
at ai.langstream.runtime.agent.AgentRunner.runJavaAgent(AgentRunner.java:268)
at ai.langstream.runtime.agent.AgentRunner.run(AgentRunner.java:191)
at ai.langstream.runtime.agent.AgentRunnerStarter.start(AgentRunnerStarter.java:120)
at ai.langstream.runtime.agent.AgentRunnerStarter.main(AgentRunnerStarter.java:55)
at ai.langstream.runtime.Main.main(Main.java:42)
20:52:37.330 [main] INFO a.l.runtime.agent.AgentRunner -- Source: TopicConsumerSource{consumer=ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader$1$1@95bb2a2}
20:52:37.330 [main] INFO a.l.runtime.agent.AgentRunner -- Processor: ai.langstream.runtime.agent.simple.IdentityAgentProvider$IdentityAgentCode@46d567cb
20:52:37.330 [main] INFO a.l.runtime.agent.AgentRunner -- Sink: ai.langstream.api.runner.code.AgentCodeAndLoader$2@363a3d15
20:52:37.330 [main] INFO a.l.runtime.agent.AgentRunner -- Service: null
20:52:37.915 [pulsar-client-io-1-1] INFO o.a.p.client.impl.ConnectionPool -- [[id: 0xea3b7ef6, L:/10.130.6.32:42032 - R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650]] Connected to server
20:52:37.950 [pulsar-client-io-1-1] INFO o.a.p.client.impl.PulsarClientImpl -- Configuring schema for topic public/default/input-topic : {
"name": "public/default/input-topic",
"schema": "",
"type": "STRING",
"timestamp": 0,
"properties": {}
}
20:52:38.109 [pulsar-client-io-1-1] INFO o.a.p.c.i.schema.AutoConsumeSchema -- Configure topic schema for topic public/default/input-topic :
20:52:38.211 [pulsar-client-io-1-1] INFO o.a.p.c.i.ConsumerStatsRecorderImpl -- Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/input-topic"],"topicsPattern":null,"subscriptionName":"langstream-agent-pipeline-python-sink-1","subscriptionType":"Failover","subscriptionProperties":null,"subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"maxAcknowledgmentGroupSize":1000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{},"readCompacted":false,"subscriptionInitialPosition":"Earliest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"startPaused":false,"autoScaledReceiverQueueSizeEnabled":false,"topicConfigurations":[],"maxPendingChuckedMessage":10}
20:52:38.215 [pulsar-client-io-1-1] INFO o.a.p.c.i.ConsumerStatsRecorderImpl -- Pulsar client config: {"serviceUrl":"pulsar://dev01-proxy.dev01-neuron-pulsar.svc.cluster.local:6650","authPluginClassName":null,"authParams":null,"authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"connectionMaxIdleSeconds":180,"useTcpNoDelay":true,"useTls":false,"tlsKeyFilePath":null,"tlsCertificateFilePath":null,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":false,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"readTimeoutMs":60000,"autoCertRefreshSeconds":300,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsKeyStoreType":"JKS","tlsKeyStorePath":null,"tlsKeyStorePassword":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":67108864,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null,"description":null}
20:52:38.227 [pulsar-client-io-1-1] INFO o.a.p.client.impl.ConnectionPool -- [[id: 0x1cb67aa5, L:/10.130.6.32:42036 - R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650]] Connected to server
20:52:38.227 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ClientCnx -- [id: 0x1cb67aa5, L:/10.130.6.32:42036 - R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650] Connected through proxy to target broker at dev01-broker-2.dev01-broker.dev01-neuron-pulsar.svc.cluster.local:6650
20:52:38.231 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ConsumerImpl -- [public/default/input-topic][langstream-agent-pipeline-python-sink-1] Subscribing to topic on cnx [id: 0x1cb67aa5, L:/10.130.6.32:42036 - R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650], consumerId 0
20:52:38.236 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ConsumerImpl -- [public/default/input-topic][langstream-agent-pipeline-python-sink-1] Subscribed to topic on dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650 -- consumer: 0
20:52:38.236 [main] INFO a.l.r.agent.TopicConsumerSource -- Starting consumer ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader$1$1@95bb2a2
20:52:38.238 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Python code directory /app-code-download/python
20:52:38.819 [MainThread] INFO root -- Configuration: {"className": "example.Exclamation"}
20:52:38.819 [MainThread] INFO root -- Environment: []
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/app/python_libs/langstream_grpc/main.py", line 37, in
server = AgentServer(sys.argv[1], sys.argv[2], sys.argv[3])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/python_libs/langstream_grpc/grpc_service.py", line 406, in init
self.agent = init_agent(configuration, json.loads(context))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/python_libs/langstream_grpc/grpc_service.py", line 369, in init_agent
module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example'
20:52:39.609 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:40.610 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:41.611 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:42.611 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:43.612 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:44.613 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:45.614 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:46.615 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:47.615 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:48.616 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:49.617 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Waiting for python agent to start
20:52:50.617 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Could not start the python agent
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at ai.langstream.agents.grpc.AgentServiceGrpc$AgentServiceBlockingStub.agentInfo(AgentServiceGrpc.java:298)
at ai.langstream.agents.grpc.PythonGrpcServer.start(PythonGrpcServer.java:96)
at ai.langstream.agents.grpc.PythonGrpcAgentSink.start(PythonGrpcAgentSink.java:37)
at ai.langstream.api.runner.code.AgentCodeAndLoader.executeWithContextClassloader(AgentCodeAndLoader.java:50)
at ai.langstream.api.runner.code.AgentCodeAndLoader$2.start(AgentCodeAndLoader.java:196)
at ai.langstream.runtime.agent.AgentRunner.runMainLoop(AgentRunner.java:657)
at ai.langstream.runtime.agent.AgentRunner.runJavaAgent(AgentRunner.java:423)
at ai.langstream.runtime.agent.AgentRunner.run(AgentRunner.java:191)
at ai.langstream.runtime.agent.AgentRunnerStarter.start(AgentRunnerStarter.java:120)
at ai.langstream.runtime.agent.AgentRunnerStarter.main(AgentRunnerStarter.java:55)
at ai.langstream.runtime.Main.main(Main.java:42)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/[0:0:0:0:0:0:0:1]:38925
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
20:52:50.618 [main] INFO a.l.agents.grpc.PythonGrpcServer -- Python process exited with code 1
20:52:50.618 [main] INFO a.l.r.agent.TopicConsumerSource -- Closing consumer ai.langstream.api.runner.topics.TopicConnectionsRuntimeAndLoader$1$1@95bb2a2
20:52:50.624 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ConsumerImpl -- [public/default/input-topic] [langstream-agent-pipeline-python-sink-1] Closed consumer
20:52:50.625 [main] INFO a.l.runtime.agent.AgentRunner -- Agent fully stopped
20:52:50.626 [main] INFO org.eclipse.jetty.server.Server -- Stopped Server@4983159f{STOPPING}[11.0.15,sto=0]
20:52:50.628 [main] INFO o.e.jetty.server.AbstractConnector -- Stopped ServerConnector@e54303{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
20:52:50.629 [main] INFO o.e.j.server.handler.ContextHandler -- Stopped o.e.j.s.ServletContextHandler@7b8233cd{/,null,STOPPED}
20:52:50.631 [main] INFO o.a.p.client.impl.PulsarClientImpl -- Client closing. URL: pulsar://dev01-proxy.dev01-neuron-pulsar.svc.cluster.local:6650
20:52:50.635 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ClientCnx -- [id: 0xea3b7ef6, L:/10.130.6.32:42032 ! R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650] Disconnected
20:52:50.638 [pulsar-client-io-1-1] INFO o.a.pulsar.client.impl.ClientCnx -- [id: 0x1cb67aa5, L:/10.130.6.32:42036 ! R:dev01-proxy.dev01-neuron-pulsar.svc.cluster.local/172.30.177.37:6650] Disconnected
20:52:52.647 [main] INFO a.langstream.impl.nar.NarFileHandler -- Closing classloader NarFileClassLoader{name='langstream-pulsar-runtime-0.4.2-nar.nar'}
20:52:52.648 [main] INFO a.langstream.impl.nar.NarFileHandler -- Closing classloader NarFileClassLoader{name='langstream-agent-grpc-0.4.2-nar.nar'}
20:52:52.662 [main] INFO a.l.runtime.agent.AgentRunnerStarter -- Error, NOW SLEEPING
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at ai.langstream.agents.grpc.AgentServiceGrpc$AgentServiceBlockingStub.agentInfo(AgentServiceGrpc.java:298)
at ai.langstream.agents.grpc.PythonGrpcServer.start(PythonGrpcServer.java:96)
at ai.langstream.agents.grpc.PythonGrpcAgentSink.start(PythonGrpcAgentSink.java:37)
at ai.langstream.api.runner.code.AgentCodeAndLoader.executeWithContextClassloader(AgentCodeAndLoader.java:50)
at ai.langstream.api.runner.code.AgentCodeAndLoader$2.start(AgentCodeAndLoader.java:196)
at ai.langstream.runtime.agent.AgentRunner.runMainLoop(AgentRunner.java:657)
at ai.langstream.runtime.agent.AgentRunner.runJavaAgent(AgentRunner.java:423)
at ai.langstream.runtime.agent.AgentRunner.run(AgentRunner.java:191)
at ai.langstream.runtime.agent.AgentRunnerStarter.start(AgentRunnerStarter.java:120)
at ai.langstream.runtime.agent.AgentRunnerStarter.main(AgentRunnerStarter.java:55)
at ai.langstream.runtime.Main.main(Main.java:42)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/[0:0:0:0:0:0:0:1]:38925
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Nov 06, 2023 8:53:21 PM io.grpc.internal.ManagedChannelImpl$2 uncaughtException
SEVERE: [Channel<1>: (localhost:38925)] Uncaught exception in the SynchronizationContext. Panic!
java.lang.NoClassDefFoundError: io/grpc/internal/InternalSubchannel$4
at io.grpc.internal.InternalSubchannel.updateAddresses(InternalSubchannel.java:346)
at io.grpc.internal.ManagedChannelImpl$SubchannelImpl.updateAddresses(ManagedChannelImpl.java:2093)
at io.grpc.internal.PickFirstLoadBalancer.acceptResolvedAddresses(PickFirstLoadBalancer.java:75)
at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.tryAcceptResolvedAddresses(AutoConfiguredLoadBalancerFactory.java:142)
at io.grpc.internal.ManagedChannelImpl$NameResolverListener$1NamesResolved.run(ManagedChannelImpl.java:1852)
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
at io.grpc.internal.ManagedChannelImpl$NameResolverListener.onResult(ManagedChannelImpl.java:1866)
at io.grpc.internal.RetryingNameResolver$RetryingListener.onResult(RetryingNameResolver.java:98)
at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:333)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: io.grpc.internal.InternalSubchannel$4
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at ai.langstream.impl.nar.NarFileHandler$NarFileClassLoader.findClass(NarFileHandler.java:140)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 13 more

Exception in thread "grpc-default-executor-0" java.lang.NoClassDefFoundError: io/grpc/internal/ManagedChannelImpl$SubchannelImpl$1ShutdownSubchannel
at io.grpc.internal.ManagedChannelImpl$SubchannelImpl.shutdown(ManagedChannelImpl.java:2030)
at io.grpc.internal.PickFirstLoadBalancer.shutdown(PickFirstLoadBalancer.java:126)
at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.shutdown(AutoConfiguredLoadBalancerFactory.java:164)
at io.grpc.internal.ManagedChannelImpl.shutdownNameResolverAndLoadBalancer(ManagedChannelImpl.java:381)
at io.grpc.internal.ManagedChannelImpl.panic(ManagedChannelImpl.java:849)
at io.grpc.internal.ManagedChannelImpl$2.uncaughtException(ManagedChannelImpl.java:187)
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:97)
at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
at io.grpc.internal.ManagedChannelImpl$NameResolverListener.onResult(ManagedChannelImpl.java:1866)
at io.grpc.internal.RetryingNameResolver$RetryingListener.onResult(RetryingNameResolver.java:98)
at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:333)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: io.grpc.internal.ManagedChannelImpl$SubchannelImpl$1ShutdownSubchannel
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at ai.langstream.impl.nar.NarFileHandler$NarFileClassLoader.findClass(NarFileHandler.java:140)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 14 more
20:53:52.662 [main] ERROR a.l.runtime.agent.AgentRunnerStarter -- Unexpected error
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at ai.langstream.agents.grpc.AgentServiceGrpc$AgentServiceBlockingStub.agentInfo(AgentServiceGrpc.java:298)
at ai.langstream.agents.grpc.PythonGrpcServer.start(PythonGrpcServer.java:96)
at ai.langstream.agents.grpc.PythonGrpcAgentSink.start(PythonGrpcAgentSink.java:37)
at ai.langstream.api.runner.code.AgentCodeAndLoader.executeWithContextClassloader(AgentCodeAndLoader.java:50)
at ai.langstream.api.runner.code.AgentCodeAndLoader$2.start(AgentCodeAndLoader.java:196)
at ai.langstream.runtime.agent.AgentRunner.runMainLoop(AgentRunner.java:657)
at ai.langstream.runtime.agent.AgentRunner.runJavaAgent(AgentRunner.java:423)
at ai.langstream.runtime.agent.AgentRunner.run(AgentRunner.java:191)
at ai.langstream.runtime.agent.AgentRunnerStarter.start(AgentRunnerStarter.java:120)
at ai.langstream.runtime.agent.AgentRunnerStarter.main(AgentRunnerStarter.java:55)
at ai.langstream.runtime.Main.main(Main.java:42)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/[0:0:0:0:0:0:0:1]:38925
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
20:53:52.663 [Thread-0] INFO a.l.runtime.agent.AgentRunnerStarter -- Beginning clean shutdown
@eolivelli
Copy link
Member

eolivelli commented Nov 7, 2023 •
edited
Loading

Are you using Pulsar ?

Can you please share more context ?

We can chat on Slack
https://langstream.slack.com/ssb/redirect

@eolivelli
Copy link
Member 

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/app/python_libs/langstream_grpc/main.py", line 37, in
server = AgentServer(sys.argv[1], sys.argv[2], sys.argv[3])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/python_libs/langstream_grpc/grpc_service.py", line 406, in init
self.agent = init_agent(configuration, json.loads(context))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/python_libs/langstream_grpc/grpc_service.py", line 369, in init_agent
module = importlib.import_module(module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example'

Can you please share your directory structure ?

This is a quick guide about writing custom agents
https://docs.langstream.ai/pipeline-agents/custom-agents/agent-developer-guide

Here you can find some examples
https://github.com/LangStream/langstream/tree/main/examples/applications/python

@arminwoworsky
Copy link
Author

codestorage section was missing in our values.yaml.

@eolivelli
Copy link
Member

@arminwoworsky you have fallen into this problem
#542

can you please write here your solution for reference ?

Unfortunately we haven't published docs about the helm chart yet

@eolivelli eolivelli changed the title Issue with running simple python example on our remote dev environemnt Issue with running simple python example on our remote dev environment on k8s (python directory seems empty) Nov 7, 2023
This was referenced Nov 7, 2023
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eolivelli @arminwoworsky