Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Java 11 causes Error injecting constructor, java.io.IOException: Can not attach to current VM #752

Closed
chancez opened this issue May 12, 2019 · 13 comments
Closed

Java 11 causes Error injecting constructor, java.io.IOException: Can not attach to current VM #752

chancez opened this issue May 12, 2019 · 13 comments
Milestone
312

Comments

@chancez
Copy link
Member

chancez commented May 12, 2019

Getting the following error when trying to run Presto 310 with Java 11. I also found the same issue in the prestosql slack channel https://prestosql.slack.com/archives/CFLB9AMBN/p1554760462185400.

2019-05-12T04:24:51.779Z	ERROR	main	io.prestosql.server.PrestoServer	Unable to create injector, see the following errors:

1) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.airlift.event.client.HttpEventModule.configure(HttpEventModule.java:43)

2) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.prestosql.server.ServerMainModule.setup(ServerMainModule.java:232)

2 errors
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.airlift.event.client.HttpEventModule.configure(HttpEventModule.java:43)

2) Error injecting constructor, java.io.IOException: Can not attach to current VM
  at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:41)
  at io.airlift.jmx.JmxModule.configure(JmxModule.java:55)
  while locating io.airlift.jmx.JmxAgent9
  while locating io.airlift.jmx.JmxAgent
    for the 1st parameter of io.airlift.jmx.JmxModule$JmxAnnouncementProvider.setJmxAgent(JmxModule.java:68)
  while locating io.airlift.jmx.JmxModule$JmxAnnouncementProvider
  while locating io.airlift.discovery.client.ServiceAnnouncement annotated with @com.google.inject.internal.Element(setName=,uniqueId=331, type=MULTIBINDER, keyType=)
  while locating java.util.Set<io.airlift.discovery.client.ServiceAnnouncement>
    for the 2nd parameter of io.airlift.discovery.client.Announcer.<init>(Announcer.java:68)
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:64)
  while locating io.airlift.discovery.client.Announcer
    for the 2nd parameter of io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  at io.airlift.discovery.client.DiscoveryModule.createMergingServiceSelectorFactory(DiscoveryModule.java:122)
  while locating io.airlift.discovery.client.MergingServiceSelectorFactory
  at io.airlift.discovery.client.DiscoveryModule.configure(DiscoveryModule.java:73)
  while locating io.airlift.discovery.client.ServiceSelectorFactory
    for the 1st parameter of io.airlift.discovery.client.ServiceSelectorProvider.setServiceSelectorFactory(ServiceSelectorProvider.java:50)
  at io.prestosql.server.ServerMainModule.setup(ServerMainModule.java:232)

2 errors
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:178)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240)
	at io.prestosql.server.PrestoServer.run(PrestoServer.java:120)
	at io.prestosql.server.PrestoServer.main(PrestoServer.java:70)
Caused by: java.io.IOException: Can not attach to current VM
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.<init>(HotSpotVirtualMachine.java:75)
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:57)
	at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
	at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
	at io.airlift.jmx.JmxAgent9.<init>(JmxAgent9.java:56)
	at io.airlift.jmx.JmxAgent9$$FastClassByGuice$$52a284ef.newInstance(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
	at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
	at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
	at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
	at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
	at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198)
	at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory$1.call(InternalProviderInstanceBindingImpl.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
	at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:115)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
	at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
	at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
	at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.access$300(InternalProviderInstanceBindingImpl.java:139)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory$1.call(InternalProviderInstanceBindingImpl.java:169)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
	at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:164)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
	at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82)
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
	at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:101)
	at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:245)
	at com.google.inject.internal.Initializer.injectAll(Initializer.java:140)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
	... 5 more
@electrum
Copy link
Member

Try adding this to etc/jvm.config:

-Djdk.attach.allowAttachSelf=true

If that's the problem, we should add that to the deployment instructions.

@chancez
Copy link
Member Author

chancez commented May 12, 2019

So I tried that, and now I'm getting

1) Error injecting constructor, com.sun.tools.attach.AttachOperationFailedException: java.lang.RuntimeException: Invalid agent state: Agent already started

Note, I am running with

    -javaagent:/opt/jmx_exporter/jmx_exporter.jar=8082:/opt/jmx_exporter/config/config.yml

using https://github.com/prometheus/jmx_exporter

@chancez
Copy link
Member Author

chancez commented May 12, 2019

Trying with a minimal set of flags to figure out if it's anything specific.

@chancez
Copy link
Member Author

chancez commented May 12, 2019

It seems like these are the minimum properties needed to cause this problem:

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=8081
    -Dcom.sun.management.jmxremote.authenticate=false
    -Djdk.attach.allowAttachSelf=true

@chancez
Copy link
Member Author

chancez commented May 12, 2019 •
edited

Without any extra jvm flags, eg:
config.properties:

http-server.http.port=8080
coordinator=true
discovery-server.enabled=true
discovery.uri=http://presto:8080
node-scheduler.include-coordinator=true

jvm.config:

-server

it causes the original error, and with -Djdk.attach.allowAttachSelf=true it works. So I can't seem to enable JMX at all currently, and even without JMX enabled, the allowAttachSelf must be true currently.

@electrum
Copy link
Member

The problem is that the first three flags tell the JVM to start the agent, then the Airlift code in JmxAgent9 unconditionally tries to start the agent itself. The code for Java 8 in JmxAgent8 detects that it's already running. I'm guessing there were problems doing this detection for Java 9.

We either need to fix this detection, or add a config flag to skip loading the agent. This flag might need to go in Presto to conditionally add the Guice module JmxModule, since that module assumes the JMX agent is available (it publishes a JMX announcement in discovery).

@chancez
Copy link
Member Author

chancez commented May 12, 2019

I see, so it shouldn't occur if I set the jmx options in presto only?

Also, what about jdk.attach.allowAttachSelf=true is that just kinda a default option needed since JMX is started unconditionally?

@electrum
Copy link
Member

electrum commented May 12, 2019 •
edited

jdk.attach.allowAttachSelf is what allows the Airlift JXM agent code to connect to the current JVM and start the JMX agent. It relaxes a security check in the JVM.

@chancez
Copy link
Member Author

chancez commented May 12, 2019

I see, that makes sense.

So status update for anyone else who runs into these issues:

I was able to still use JMX by only using

    jmx.rmiserver.port=8081
    jmx.rmiregistry.port=8081

in config.properties, and

    -javaagent:/opt/jmx_exporter/jmx_exporter.jar=8082:/opt/jmx_exporter/config/config.yml
    # this fixes https://github.com/prestosql/presto/issues/752
    -Djdk.attach.allowAttachSelf=true

In jvm.properties.

All seems to be working, including my Prometheus JMX exporter.

Once you touch com.sun.management.jmxremote is when things stop working, for the reasons @electrum already mentioned.

@electrum
Copy link
Member

Thanks for helping us find this confusing scenario. I have a patch to add a hint about the self-attach flag and to skip starting the agent if it's already configured: airlift/airlift#733

@electrum electrum added this to the 312 milestone Jun 27, 2019
@electrum
Copy link
Member

This was fixed in Presto 312.

@codekaust
Copy link 

-Djdk.attach.allowAttachSelf=true

This solved the issue for me @electrum, if that's the case for most of us, please add it to deployment documentation.

@findepi
Copy link
Member

findepi commented Jun 1, 2020

@codekaust thanks. This is already part of the docs -- https://prestosql.io/docs/current/installation/deployment.html#jvm-config
Please let me know if you see other places in docs which would require an update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
312
Development

No branches or pull requests
4 participants
@electrum @findepi @chancez @codekaust