-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add load balancing between available apic's #1
Comments
Hi, |
Hi @ahmedaall. I have not done any development on this issue, but If you like to contribute it would be great. Another option is to use some external LB that take care of this and that support more options like health checks etc. |
Hi @thenodon. During my failover test I saw after shutting down my Apic1 that the exporter become completely unreachable (while when i cut the apic2 or 3 the exporter still works perfectly). So instead of having my 3 Apics adresses in my config, I put the Load Balancer adresse that round robin between them with session persistence : # Profiles for different fabrics
fabrics:
# This is the Cisco provided sandbox that is open for testing
# cisco_sandbox:
# username: admin
# password: <check the cisco sandbox to get the password>
# apic:
# - https://sandboxapicdc.cisco.com
MY_FABRIC:
username: ACI_USERNAME
password: ACI_PASSWORD
apic:
- https://LB
# - https://APIC1
# - https://APIC2
# - https://APIC3 So I tried again my failover test APIC redundancy by shutting down the Apic1. And at this moment I have this aci_up metrics to DOWN : # HELP aci_up The connection state 1=UP, 0=DOWN
# TYPE aci_up gauge
aci_up{fabric="MY_FABRIC"} 0
PS : The Load Balancer round robin between APICs properly with a browser access during the failover, but the exporter still goes down. |
Hi @ahmedaall , great that you are testing this. I do not have the have an environment where I can test this myself. I think there should be something in the logs that could help. Great if you could attach. You could also do some debugging. I think a break point in aci-connection.go in the login function is a good starting point since its called on every request. |
Hi @thenodon. Yes, here are the logs after noticing that the exporter goes down only when apic1 is down. BEFORE SHUTTING DOWN APIC1 :
{"class":"topSystem","exec_time":1194174,"fabric":"MY_FABRIC","length":54981,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/topSystem.json?rsp-subtree-include=health"}
{"class":"fvTenant","exec_time":1194421,"fabric":"MY_FABRIC","length":2164,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/fvTenant.json?rsp-subtree-include=health,required"}
{"class":"faults","exec_time":1195618,"fabric":"MY_FABRIC","length":2879,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/faultCountsWithDetails.json"}
{"class":"fvCtx","exec_time":1196088,"fabric":"MY_FABRIC","length":7894,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/fvCtx.json?rsp-subtree-include=health,required"}
{"class":"eqptTemp5min","exec_time":1206707,"fabric":"MY_FABRIC","length":21586,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/eqptTemp5min.json?rsp-subtree-include=stats\u0026rsp-subtree-class=eqptTemp5min\u0026query-target-filter=wcard(eqptTemp5min.dn,\".*sup/sensor-2/CDeqptTemp5min\")"}
{"class":"fvAp","exec_time":1210511,"fabric":"MY_FABRIC","length":3642,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/fvAp.json?rsp-subtree-include=health,required"}
{"class":"fabricHealthTotal","exec_time":1211007,"fabric":"MY_FABRIC","length":544,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/fabricHealthTotal.json?query-target-filter=wcard(fabricHealthTotal.dn,\"topology/.*/health\")"}
{"class":"procSysMem5min","exec_time":1211795,"fabric":"MY_FABRIC","length":26934,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:44Z","uri":"https://APIC2/api/class/procSysMem5min.json"}
{"exec_time":2271430,"fabric":"MY_FABRIC","level":"info","msg":"total scrape time ","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","time":"2023-08-21T06:58:44Z"}
{"exec_time":35451,"fabric":"MY_FABRIC","level":"info","method":"POST","msg":"api call fabric","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:45Z","uri":"https://APIC2/api/aaaLogout.xml"}
{"exec_time":4286804,"fabric":"MY_FABRIC","length":1064980,"level":"info","method":"GET","msg":"api call","requestid":"2UHeqJJd39oafNMVwa2mHNJU3AN","status":200,"time":"2023-08-21T06:58:46Z","uri":"/probe?target=MY_FABRIC"}
AFTER SHUTTING DOWN APIC1 :
{"exec_time":90839,"fabric":"MY_FABRIC","level":"info","method":"POST","msg":"api call fabric","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","status":200,"time":"2023-08-21T07:04:31Z","uri":"https://APIC2/api/aaaLogin.xml"}
{"fabric":"MY_FABRIC","level":"info","msg":"Using apic https://APIC2","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","time":"2023-08-21T07:04:31Z"}
{"class":"aci_name","exec_time":90723882,"fabric":"MY_FABRIC","length":0,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","status":503,"time":"2023-08-21T07:06:01Z","uri":"https://APIC2/api/mo/topology/pod-1/node-1/av.json"}
{"fabric":"MY_FABRIC","level":"error","msg":"Request /api/mo/topology/pod-1/node-1/av.json failed - ACI api returned 503.","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","time":"2023-08-21T07:06:01Z"}
{"exec_time":25825,"fabric":"MY_FABRIC","level":"info","method":"POST","msg":"api call fabric","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","status":200,"time":"2023-08-21T07:06:01Z","uri":"https://APIC2/api/aaaLogout.xml"}
{"exec_time":90841072,"fabric":"MY_FABRIC","length":101,"level":"info","method":"GET","msg":"api call","requestid":"2UHfYFFTFXy9vN7M48szPfQ8cUD","status":503,"time":"2023-08-21T07:06:01Z","uri":"/probe?target=MY_FABRIC"} Looks like the MO (Managed object) is reachable only through the apic1 |
@ahmedaall Why do you not test to run against apic2 without LB and verify if that apic node works for queries. 503 is an interesting response. Running directly against apic2 from aci-exporter hopefully revile if the problem is on the apic or exporter side . |
@thenodon Yes, you have a point. I tried to run against apic2 and 3 as targets. It works perfectly. But when I shutdown the apic1 the exporter goes down : {"class":"aci_name","exec_time":90295767,"fabric":"MY_FABRIC","length":0,"level":"info","method":"GET","msg":"api call fabric","requestid":"2UHqtP9LRXobqtpFKzcGsaTZIrK","status":503,"time":"2023-08-21T08:39:18Z","uri":"https://APIC2/api/mo/topology/pod-1/node-1/av.json"}
{"fabric":"MY_FABRIC","level":"error","msg":"Request /api/mo/topology/pod-1/node-1/av.json failed - ACI api returned 503.","requestid":"2UHqtP9LRXobqtpFKzcGsaTZIrK","time":"2023-08-21T08:39:18Z"}
{"exec_time":26211,"fabric":"MY_FABRIC","level":"info","method":"POST","msg":"api call fabric","requestid":"2UHqtP9LRXobqtpFKzcGsaTZIrK","status":200,"time":"2023-08-21T08:39:18Z","uri":"https://APIC2/api/aaaLogout.xml"}
{"exec_time":90442276,"fabric":"MY_FABRIC","length":101,"level":"info","method":"GET","msg":"api call","requestid":"2UHqtP9LRXobqtpFKzcGsaTZIrK","status":503,"time":"2023-08-21T08:39:18Z","uri":"/probe?target=MY_FABRIC"} It look like an APIC issue |
Do you have the same behavior just using curl? Like to take aci-exporter out of the equation :) |
The curl works from the exporter to the APICs :) |
@ahmedaall I meant do curl works directly at the apic2 when you take apic1 down for queries? |
@thenodon yes, I confirm that curl works from the exporter to the apic2 when apic1 is down |
@ahmedaall as I interpret your answer is that the problem is related to the LB or aci-exporter, since curl request to any apic will continue to work even if one, or more apics, are down. A couple of more questions. |
@thenodon all the tests were done by directly specifying the target apics as below : MY_FABRIC:
username: ACI_USERNAME
password: ACI_PASSWORD
apic:
# - https://LB
- https://APIC2
- https://APIC3
# - https://APIC1 I did not debug by putting the load balancer because I assume that the problem would not be on this side. But I confirm you that curl from the exporter to the APICs and LB works properly. |
Sound really strange - aci-exporter connect to https://APIC2 and that works until APIC1 is shutdown - correct? But using curl against https://APIC2 works even after APIC1 is shutdown. Is this correct ? |
Exactly |
the only explanation would be that there is a dependency towards apic1. In the logs we see that he manages to login but can't access https://APIC/api/mo/.../.../... It's unlikely but maybe access to the MO is only possible through the APIC1. To verify this hypothesis I will try to make a terraform plan from one of my vm by first turning off the APIC1 |
But how can it work for curl ? |
@camrossi don you any ideas what this problem can be related to? |
Maybe there is a policy or other things that block the access against the MO, but not against the rest. curl https://APIC2 |
I tried to do terraform plan while shutting down apic1 to see if it is a spof for api access. It works.... So maybe the problem is in the exporter side. |
Can you tell me more about the "terraform plan" you did. |
I wanted to verify the hypothesis that only apic1 can communicate with the api. I therefore launched a terraform plan against the address of the load balancer while apic1 was off. And it worked.. |
Hi @thenodon, I found the issue ! urlMap := make(map[string]string)
urlMap["login"] = "/api/aaaLogin.xml"
urlMap["logout"] = "/api/aaaLogout.xml"
urlMap["faults"] = "/api/class/faultCountsWithDetails.json"
urlMap["aci_name"] = "/api/mo/topology/pod-1/node-1/av.json" The urlMap aci_name contains only the path to the "pod-1/node-1" which corresponds to...APIC1. It is therefore normal that with each reboot of the APIC1, the export becomes unavailable. I think aci-connection.go should be updated so that it can contain the url maps of all the APICs targets entered in the config file. |
@ahmedaall - thanks for the trouble shooting. So if I have 2 apic you have node-1 and node-2? Is it possible to get the number of apic nodes? How is node numbering defined in the apic? node-1 will be the first to boot or is it a configuration? |
@thenodon How is node numbering defined in the apic? node-1 will be the first to boot or is it a configuration? |
@ahmedaall I have tried to solve this issue. Would be great if you can test it. You can build aci-exporter from branch https://github.com/opsdis/aci-exporter/tree/issue_1. The fix is in commit da40506. |
Hi @thenodon. Thank you for the update ! I'll keep you inform when update my exporter |
@thenodon I don't understand what value I have to enter in aci_name value |
@ahmedaall its optional, but if you set it the name from the ACI will not be used. This is the value of the |
@ahmedaall any update on this? |
Hi Folks, I also tested your code and is not crashing for me. I implemented this as a test BUT I do not know Go, I am not joking this is the first time I write anything in go so is probably horrible and I am amazed it even works
|
@camrossi - glad to here that the branch worked for you. And your golang works fine. |
I build from golang:latest. I am using version go1.21.1, in a Debian 12 |
@ahmedaall I have the same golang version and running on ubuntu 22.04. I would suggest that you test what I recommended in #1 (comment) |
@thenodon I just launch the sandbox. I am a newbie with it. I'll do the test as fast as possible. |
The cisco sandbox is on the internet, https://devnetsandbox.cisco.com/RM/Topology, so just configure aci-exporter with:
|
@thenodon ok I'll take those parameters and for the minimal config I'll directly import example-config.yaml and desactivate my current config yaml file |
Done. But what url do I need to enter to launch the exporter with sandbox target ? |
@ahmedaall if you used the above config för the cisco sandbox the name is
Where |
@thenodon Everything works fine with the sandbox : {"level":"info","msg":"aci-exporter starting on port 9643","time":"2023-09-14T14:04:14Z"} {"level":"info","msg":"Read timeout 0s, Write timeout 0s","time":"2023-09-14T14:04:14Z"} {"exec_time":1015415,"fabric":"cisco_sandbox","level":"info","method":"POST","msg":"api call fabric","requestid":"2VOI1DmZOBkw9N0AUieRthm53KK","status":200,"time":"2023-09-14T14:08:03Z","uri":"https://sandboxapicdc.cisco.com/api/aaaLogin.xml"} {"fabric":"cisco_sandbox","level":"info","msg":"Using apic https://sandboxapicdc.cisco.com","requestid":"2VOI1DmZOBkw9N0AUieRthm53KK","time":"2023-09-14T14:08:03Z"} {"class":"fabricNode","exec_time":189166,"fabric":"cisco_sandbox","length":580,"level":"info","method":"GET","msg":"api call fabric","requestid":"2VOI1DmZOBkw9N0AUieRthm53KK","status":200,"time":"2023-09-14T14:08:03Z","uri":"https://sandboxapicdc.cisco.com/api/class/fabricNode.json?query-target-filter=eq(fabricNode.role,\"controller\")"} |
@ahmedaall can you run the same config with the same query against your own apic. Just add you config in the fabric section of the config file |
@thenodon So in the same file I change the target from the sandbox to my APIC and I have this : {"level":"info","msg":"aci-exporter starting on port 9643","time":"2023-09-14T14:35:03Z"} |
@ahmedaall so now against the lb?
First try to add the
Do you get the same panic? You should also verify to run against one of the apic endpoint, no lb. |
@thenodon |
@ahmedaall so everything works with the released version, both through lb and against apic endpoint. |
@thenodon I have version 5.2(6e) |
I tested against a real ACI 5.2(8) with my code and it worked just fine by using
but this works just fine, can you check if you have the same behaviour ?
@thenodon let me see if I can get you read only access to one of our DMZ fabrics so you can actually test on something real, might not have lots of flexibility in term of versions but would be better than what you have now :D Here my crash trace when the uppercase:
|
If I make |
@camrossi thanks for your findings. I can confirm I can reproduce the same behavior with the uppercase name of the fabric. Currently not sure why, but I think its a combination on how yaml package deserialize the yaml file and how I match the target query path against the deserialize fabrics structures. I will investigate it more and what changed between the released version and the branch version related to dependency since @ahmedaall got it to work in the released version. |
@camrossi @thenodon thank you for the debug. So I try to change from uppercase to lower for the name of the fabric : # Exporter port
port: 9643
# Configuration file name default without postfix
config: config
# The prefix of the metrics
prefix:
fabrics:
# cisco_sandbox:
# username: admin
# password: ""
# apic:
# - https://sandboxapicdc.cisco.com
my_fabric:
username: ACI_USERNAME
password: ACI_PASSWORD
apic:
- https://APIC1
- https://APIC2
- https://APIC3 but...I have the same issue. I try with and without the lb. Did I forgot something ? |
Yes @ahmedaall you need to update the prometheus config as well (assuming you use that) so that it uses the lowercase name.
and then in aci-exporter
|
@camrossi Yes I didn't mention it but I did update my target in prometheus file :) |
@ahmedaall @camrossi have added documentation and check on /probe endpoint that fabric is in lower case. You can do a pull on the issue_1 branch. For more info see commit 6924973 |
@ahmedaall have you had a chance to verify your LB solution yet? Would like to close this and make a release. |
@thenodon not yet... i'll do it today or tomorrow. Sorry for the time it takes. |
@thenodon Ok I just finished my failover test. Everything works perfectly. Even if I shutdown 2 of my 3 APICs it works good ! |
Great @ahmedaall. If you have the time to share your setup I can add it to the README. Hope you are now happy with aci-exporter and give it a star. |
@thenodon I confirm you that I am very happy with the aci-exporter. It is a really good job. Thanks a lot your reactivity with this issue. |
Thanks @ahmedaall and @camrossi for your support. Will close this with pull request #38 |
Describe the solution you'd like
If more then one apic is configured for the fabric the exporter should be able to round-robin between them.
The text was updated successfully, but these errors were encountered: