Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Azure ARM discovery plugin #22679

Closed
wants to merge 29 commits into from
Closed

Conversation

@dadoonet
Copy link
Member

@dadoonet dadoonet commented Jan 18, 2017

Supported settings so far:

cloud:
    azure-arm:
        client_id: FILL_WITH_YOUR_CLIENT_ID
        secret: FILL_WITH_YOUR_SECRET
        tenant_id: FILL_WITH_YOUR_TENANT
        subscription_id: FILL_WITH_YOUR_SUBSCRIPTION_ID

discovery:
    zen.hosts_provider: azure-arm
    azure-arm:
        host:
            type: private_ip
            name: azure-esnode-master-*
            group_name: azure-preprod
            region: westeurope
        refresh_interval: 10s

Closes #19146

* Azure ARM secret
*/
public static final Setting<String> SECRET_SETTING =
Setting.simpleString("cloud.azure-arm.secret", Property.NodeScope, Property.Filtered);

This comment has been minimized.

@rjernst

rjernst Jan 18, 2017
Member

I don't think we should be adding new secret settings that are plain strings.

This comment has been minimized.

@dadoonet

dadoonet Jan 18, 2017
Author Member

Agreed.

You can filter virtual machines you would like to connect to by entering a name here. It can be a wildcard
like `azure-esnode-*`.

`discovery.azure-arm.host.group_name`::

This comment has been minimized.

@Mpdreamz

Mpdreamz Jan 19, 2017
Member

Can we rename this to discovery.azure-arm.resource_group, group by itself is a tad ambiguous and resource group is a well understood azure concept

This comment has been minimized.

@dadoonet

dadoonet Jan 19, 2017
Author Member

Sure! Will do!

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Jan 20, 2017

@Mpdreamz I updated the repo with new changes:

  • Fixed the missing lib issue
  • Renamed group_name to resource_group
  • Fixed an issue when using a wildcard in resource_group
@Mpdreamz
Copy link
Member

@Mpdreamz Mpdreamz commented Jan 20, 2017

Now getting security manager exceptions @dadoonet

[2017-01-20T12:52:57,233][WARN ][o.e.d.z.ZenDiscovery ] [data-0] Ping execution failed

│java.util.concurrent.ExecutionException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")

@jasontedor
Copy link
Member

@jasontedor jasontedor commented Jan 20, 2017

We need the entire stack trace here.

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Jan 20, 2017

@Mpdreamz Interesting. Can you start elasticsearch with this option -Djava.security.debug="access,failure"?

ES_JAVA_OPTS='-Djava.security.debug="access,failure"' ./bin/elasticsearch
@Mpdreamz
Copy link
Member

@Mpdreamz Mpdreamz commented Jan 20, 2017

My bad @jasontedor: click to expand

[2017-01-20T12:52:57,233][WARN ][o.e.d.z.ZenDiscovery ] [data-0] Ping execution failed
java.util.concurrent.ExecutionException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_121]
at org.elasticsearch.discovery.zen.ZenDiscovery.pingAndWait(ZenDiscovery.java:1000) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:860) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:372) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.ZenDiscovery.access$3800(ZenDiscovery.java:80) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1176) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
at java.lang.Class.checkMemberAccess(Class.java:2348) ~[?:1.8.0_121]
at java.lang.Class.getEnclosingMethod(Class.java:1037) ~[?:1.8.0_121]
at sun.reflect.generics.scope.ClassScope.computeEnclosingScope(ClassScope.java:50) ~[?:?]
at sun.reflect.generics.scope.AbstractScope.getEnclosingScope(AbstractScope.java:78) ~[?:?]
at sun.reflect.generics.scope.AbstractScope.lookup(AbstractScope.java:96) ~[?:?]
at sun.reflect.generics.factory.CoreReflectionFactory.findTypeVariable(CoreReflectionFactory.java:110) ~[?:?]
at sun.reflect.generics.visitor.Reifier.visitTypeVariableSignature(Reifier.java:165) ~[?:?]
at sun.reflect.generics.tree.TypeVariableSignature.accept(TypeVariableSignature.java:43) ~[?:?]
at sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68) ~[?:?]
at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138) ~[?:?]
at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49) ~[?:?]
at sun.reflect.generics.repository.ClassRepository.getSuperclass(ClassRepository.java:90) ~[?:?]
at java.lang.Class.getGenericSuperclass(Class.java:777) ~[?:1.8.0_121]
at com.fasterxml.jackson.core.type.TypeReference.(TypeReference.java:33) ~[jackson-core-2.8.6.jar:2.8.6]
at com.microsoft.rest.serializer.JacksonMapperAdapter$1.(JacksonMapperAdapter.java:179) ~[?:?]
at com.microsoft.rest.serializer.JacksonMapperAdapter.deserialize(JacksonMapperAdapter.java:179) ~[?:?]
at com.microsoft.rest.ServiceResponseBuilder.buildBody(ServiceResponseBuilder.java:289) ~[?:?]
at com.microsoft.rest.ServiceResponseBuilder.build(ServiceResponseBuilder.java:141) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.listDelegate(VirtualMachinesInner.java:1101) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.access$600(VirtualMachinesInner.java:46) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner$36.call(VirtualMachinesInner.java:1091) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner$36.call(VirtualMachinesInner.java:1087) ~[?:?]
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:173) ~[?:?]
at rx.Subscriber.setProducer(Subscriber.java:211) ~[?:?]
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:9861) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.Observable.subscribe(Observable.java:9957) ~[?:?]
at rx.Observable.subscribe(Observable.java:9924) ~[?:?]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445) ~[?:?]
at rx.observables.BlockingObservable.single(BlockingObservable.java:342) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.list(VirtualMachinesInner.java:1006) ~[?:?]
at com.microsoft.azure.management.compute.implementation.VirtualMachinesImpl.listByGroup(VirtualMachinesImpl.java:68) ~[?:?]
at org.elasticsearch.cloud.azure.arm.AzureManagementServiceImpl.lambda$getVirtualMachines$1(AzureManagementServiceImpl.java:103) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
at org.elasticsearch.cloud.azure.arm.AzureManagementServiceImpl.getVirtualMachines(AzureManagementServiceImpl.java:92) ~[?:?]
at org.elasticsearch.discovery.azure.arm.AzureArmUnicastHostsProvider.buildDynamicNodes(AzureArmUnicastHostsProvider.java:100) ~[?:?]
at org.elasticsearch.discovery.zen.UnicastZenPing.ping(UnicastZenPing.java:302) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.UnicastZenPing.ping(UnicastZenPing.java:279) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
at org.elasticsearch.discovery.zen.ZenDiscovery.pingAndWait(ZenDiscovery.java:993) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
... 8 more

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Jan 20, 2017

I opened Azure/autorest-clientruntime-for-java#136 on azure side.
The problem here is that we load jackson with the core security manager which does not allow java.lang.RuntimePermission "accessDeclaredMembers" so even if I change the policy for the plugin, when Azure SDK will call Jackson, it will call it with the core security manager policy.

Let see what Azure team can do to help us in that context.

If they can't do anything, I believe that we would have to shade somehow a version of Azure SDK and Jackson within a single flat jar where we can relocate then Jackson classes... I'm not a fan of this TBH.

@rjernst
Copy link
Member

@rjernst rjernst commented Jan 20, 2017

That looks like an issue in jackson itself. The TypeReference ctor calls Class.getGenericSuperclass(), which from an anonymous class requires that permission (and the adaptor in azure creates an anonymous TypeReference instance. It seems like a web of complicated design that may not be possible to untangle (even with a flat jar).

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Jan 21, 2017

@ejsmith
Copy link

@ejsmith ejsmith commented Jan 24, 2017

Will this support scalesets?

@hglkrijger
Copy link

@hglkrijger hglkrijger commented Apr 20, 2017

@dadoonet, just wondering if this is still getting merged?

dadoonet added 4 commits Jul 19, 2016
Supported settings so far:

```yml
cloud:
    azure-arm:
        client_id: FILL_WITH_YOUR_CLIENT_ID
        secret: FILL_WITH_YOUR_SECRET
        tenant_id: FILL_WITH_YOUR_TENANT
        subscription_id: FILL_WITH_YOUR_SUBSCRIPTION_ID

discovery:
    zen.hosts_provider: azure-arm
    azure-arm:
        host:
            type: private_ip
            name: azure-esnode-master-*
            group_name: azure-preprod
            region: westeurope
        refresh_interval: 10s
```

Closes #19146
And cleanup a bit gradle build
Also adds more Javadoc on settings.
Also fixed a bug when using wildcards for group names. Azure API does not support wildcards so we need to get all the VMs in that case and filter on our side.
Applying here advices I got at Azure/azure-sdk-for-java#1387 (comment)
@dadoonet dadoonet force-pushed the dadoonet:pr/azure-arm branch to db6ea7b May 24, 2017
@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Jul 27, 2018

This is now to work well from an elasticsearch node:

[2018-07-27T15:15:16,608][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] Ignoring machine [base6/10.0.0.4] because of [DEALLOCATED] power status
[2018-07-27T15:15:16,609][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [logstash]: [10.0.1.9]
[2018-07-27T15:15:16,609][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.1.9, transport_address 10.0.1.9:9300
[2018-07-27T15:15:16,609][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [lsdata-0]: [10.0.1.6]
[2018-07-27T15:15:16,610][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.1.6, transport_address 10.0.1.6:9300
[2018-07-27T15:15:16,610][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [lsdata-1]: [10.0.1.7]
[2018-07-27T15:15:16,610][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.1.7, transport_address 10.0.1.7:9300
[2018-07-27T15:15:16,610][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [lsdata-2]: [10.0.1.8]
[2018-07-27T15:15:16,611][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.1.8, transport_address 10.0.1.8:9300
[2018-07-27T15:15:16,611][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [lskibana]: [10.0.1.5]
[2018-07-27T15:15:16,611][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.1.5, transport_address 10.0.1.5:9300
[2018-07-27T15:15:16,611][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] found networkAddress for [dpi-arm-test]: [10.0.2.4]
[2018-07-27T15:15:16,612][TRACE][o.e.d.a.a.AzureArmUnicastHostsProvider] adding 10.0.2.4, transport_address 10.0.2.4:9300
[2018-07-27T15:15:16,612][DEBUG][o.e.d.a.a.AzureArmUnicastHostsProvider] 6 hosts(s) added

Not sure why the build is not working well yet.
Let's figure this out when the review is done.

In the meantime, someone from @elastic/microsoft team would like to test this new plugin?

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Aug 6, 2018

@elasticmachine retest this please

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Aug 6, 2018

I don't know why org.elasticsearch.index.engine.InternalEngineTests.testSeqNoAndCheckpoints is failing. Does not seem related to my change.

Anyway, someone from the @elastic/es-distributed team would like to review?

@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Aug 6, 2018

Merging in latest master should fix this (see #32430)

@russcam
Copy link
Contributor

@russcam russcam commented Aug 6, 2018

@dadoonet I'd like to test this with the Azure ARM template; is it possible to build a version for 6.3.1?

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Aug 7, 2018

@ywelsch I was sure I did. And actually I did not. 😄

I think I have a trickier thing to solve now.
When I run this locally with JDK10, it passes.

./gradlew :plugins:discovery-azure-arm:thirdPartyAudit

But with JDK8, this is failing with:

08:45:41 Execution failed for task ':plugins:discovery-azure-arm:thirdPartyAudit'.
08:45:41 > Invalid exclusions, nothing is wrong with these classes: [javax/activation/ActivationDataFlavor.class, javax/activation/DataContentHandler.class, javax/activation/DataHandler.class, javax/activation/DataSource.class, javax/activation/FileDataSource.class, javax/activation/FileTypeMap.class]

(Source: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/14717/console)

Do you know how I can make this check optional depending on the JVM version? Unless that's a bad idea and that should be solved in another way?

@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Aug 7, 2018

Do you know how I can make this check optional depending on the JVM version? Unless that's a bad idea and that should be solved in another way?

yes, it should be made optional depending on JVM version, see the build file for discovery-azure-classic (or other build files).

@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Aug 7, 2018

Thanks @ywelsch that worked! Should I rebase and squash before anyone review it?

@dadoonet dadoonet requested a review from elastic/es-distributed Aug 7, 2018
@jasontedor jasontedor removed the request for review from elastic/es-distributed Aug 28, 2018
@rjernst rjernst removed the review label Oct 10, 2018
@ofer-velich
Copy link

@ofer-velich ofer-velich commented Dec 4, 2018

wondering when discovery-azure plugin should be available?
what should I do for discovery if I'm running elasticsearch on azure arm instances in the meantime?
tnx

dadoonet added 3 commits Dec 11, 2018
# Conflicts:
#	docs/reference/cat/plugins.asciidoc
#   docs/reference/modules/discovery/azure.asciidoc
# Conflicts:
#	docs/reference/cat/plugins.asciidoc
@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Mar 19, 2019

We discussed this PR during the distributed sync and, while generally useful, think that we currently cannot take this one on, in particular give it the proper testing, support and maintenance that we provide for the other plugins in the ES repo. This might be a better fit as a community plugin for now.

@ywelsch ywelsch closed this Mar 19, 2019
@dadoonet
Copy link
Member Author

@dadoonet dadoonet commented Mar 19, 2019

Can we then remove this part of the documentation? https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-azure-classic.html

image

Or fix its content saying that this plugin will be removed in the future?

@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Mar 19, 2019

sure, thank you for pointing this out.

ywelsch added a commit that referenced this pull request Mar 20, 2019
Relates to #22679
ywelsch added a commit that referenced this pull request Mar 20, 2019
Relates to #22679
ywelsch added a commit that referenced this pull request Mar 20, 2019
Relates to #22679
pgomulka added a commit to pgomulka/elasticsearch that referenced this pull request Mar 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet