Cisco: default-route originate route-map with a set-community statement crashes (setting community on generated route, not bgp route)

[kmjmartin]

Hi,

running into this error with BGP questions.
We're using 4 questions, it only fails on the 4th one:

bfq.bgpProcessConfiguration().answer().frame() <- works
bfq.bgpPeerConfiguration().answer().frame() <- works
bfq.bgpSessionCompatibility().answer().frame() <- works
bfq.bgpSessionStatus().answer().frame() <- fails with the following error:

Traceback (most recent call last):
File "batfish/device_compliance.py", line 2771, in
main(prepare_env=True, source_dir="../net_config_collector/configs")
File "batfish/device_compliance.py", line 2686, in main
bgp_configuration_audit(filename_prefix=filename_prefix)
File "batfish/device_compliance.py", line 2362, in bgp_configuration_audit
all_df_list = fetch_bgp_config_dfs()
File "batfish/device_compliance.py", line 2337, in fetch_bgp_config_dfs
bgp_session_status_df = bfq.bgpSessionStatus().answer().frame()
File "/home//.pyenv/versions/net_compliance/lib/python3.6/site-packages/pybatfish/question/question.py", line 200, in answer
extra_args=extra_args,
File "/home//.pyenv/versions/net_compliance/lib/python3.6/site-packages/pybatfish/client/internal.py", line 58, in _bf_answer_obj
workhelper.execute(work_item, session, background, extra_args)
File "/home//.pyenv/versions/net_compliance/lib/python3.6/site-packages/pybatfish/client/workhelper.py", line 107, in execute
session, CoordConsts.SVC_RSC_QUEUE_WORK, json_data
File "/home//.pyenv/versions/net_compliance/lib/python3.6/site-packages/pybatfish/client/resthelper.py", line 78, in get_json_response
"Coordinator returned failure: {}".format(json_response[1])
pybatfish.exception.BatfishException: Coordinator returned failure: Failed to queue work: Cannot queue dataplane dependent work for 9016ea02-8b76-4ff9-bfd2-bf138e5cf619: Status is DATAPLANING_FAIL but no incomplete dataplaning work exists

Please let me know what logs of config files you'd require to be able to look into this.

Thanks,

Kane

[dhalperi]

@kmjmartin Thanks! can you run bf.generate_dataplane() and see the logs there?

You can also find us in real-time on Batfish slack

[kmjmartin]

Hi @dhalperi

Thanks, here's the output:

pybatfish.exception.BatfishException: Work terminated abnormally
work_item: {"containerName": "xx-network", "id": "ceb44800-c11e-4683-8d5b-93d5c5f49b44", "requestParams": {"dp": "", "testrig": "tu-network-2021-03-05"}, "testrigName": "xx-network-2021-03-05"}

log: Loading configurations for NetworkSnapshot{network=bd73afbd-a3aa-4c44-be9e-2445512f65b0, snapshot=985c4a7e-010b-4f1b-8758-e9bbf944d15f}
Loading configurations for NetworkSnapshot{network=bd73afbd-a3aa-4c44-be9e-2445512f65b0, snapshot=985c4a7e-010b-4f1b-8758-e9bbf944d15f}
Loading configurations for NetworkSnapshot{network=bd73afbd-a3aa-4c44-be9e-2445512f65b0, snapshot=985c4a7e-010b-4f1b-8758-e9bbf944d15f}
Loading configurations for NetworkSnapshot{network=bd73afbd-a3aa-4c44-be9e-2445512f65b0, snapshot=985c4a7e-010b-4f1b-8758-e9bbf944d15f}
Loading configurations for NetworkSnapshot{network=bd73afbd-a3aa-4c44-be9e-2445512f65b0, snapshot=985c4a7e-010b-4f1b-8758-e9bbf944d15f}
Exception in container:bd73afbd-a3aa-4c44-be9e-2445512f65b0, testrig:985c4a7e-010b-4f1b-8758-e9bbf944d15f; exception:java.lang.ClassCastException: class org.batfish.datamodel.GeneratedRoute$Builder cannot be cast to class org.batfish.datamodel.BgpRoute$Builder (org.batfish.datamodel.GeneratedRoute$Builder and org.batfish.datamodel.BgpRoute$Builder are in unnamed module of loader 'app')
at org.batfish.datamodel.routing_policy.communities.SetCommunities.execute(SetCommunities.java:33)
at org.batfish.datamodel.routing_policy.RoutingPolicy.call(RoutingPolicy.java:117)
at org.batfish.datamodel.routing_policy.expr.CallExpr.evaluate(CallExpr.java:66)
at org.batfish.datamodel.routing_policy.statement.If.execute(If.java:107)
at org.batfish.datamodel.routing_policy.RoutingPolicy.call(RoutingPolicy.java:117)
at org.batfish.datamodel.routing_policy.RoutingPolicy.process(RoutingPolicy.java:244)
at org.batfish.datamodel.routing_policy.RoutingPolicy.process(RoutingPolicy.java:199)
at org.batfish.dataplane.protocols.GeneratedRouteHelper.activateGeneratedRoute(GeneratedRouteHelper.java:47)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.processNeighborSpecificGeneratedRoute(BgpRoutingProcess.java:960)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.lambda$getOutgoingRoutesForEdge$24(BgpRoutingProcess.java:867)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.base/java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:723)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.pullV4UnicastMessages(BgpRoutingProcess.java:655)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.processBgpV4UnicastMessages(BgpRoutingProcess.java:602)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.processBgpMessages(BgpRoutingProcess.java:492)
at org.batfish.dataplane.ibdp.BgpRoutingProcess.executeIteration(BgpRoutingProcess.java:412)
at org.batfish.dataplane.ibdp.VirtualRouter.bgpIteration(VirtualRouter.java:1323)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.lambda$null$15(IncrementalBdpEngine.java:437)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.lambda$computeIterationOfBgpRoutes$16(IncrementalBdpEngine.java:437)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
at java.base/java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:1957)
at java.base/java.util.concurrent.ForkJoinTask.tryExternalHelp(ForkJoinTask.java:378)
at java.base/java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:323)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:412)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeIterationOfBgpRoutes(IncrementalBdpEngine.java:434)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeDependentRoutesIteration(IncrementalBdpEngine.java:418)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeNonMonotonicPortionOfDataPlane(IncrementalBdpEngine.java:693)
at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeDataPlane(IncrementalBdpEngine.java:240)
at org.batfish.dataplane.ibdp.IncrementalDataPlanePlugin.computeDataPlane(IncrementalDataPlanePlugin.java:46)
at org.batfish.main.Batfish.computeDataPlane(Batfish.java:806)
at org.batfish.main.Batfish.run(Batfish.java:2197)
at org.batfish.main.Driver.lambda$runBatfish$0(Driver.java:312)
at java.base/java.lang.Thread.run(Thread.java:834)
{"answerElements":[{"class":"org.batfish.common.BatfishException$BatfishStackTrace","answer":["org.batfish.common.BatfishException: Batfish job failed"," at org.batfish.main.Driver.lambda$runBatfish$0(Driver.java:352)"," at java.base/java.lang.Thread.run(Thread.java:834)","Caused by: java.lang.ClassCastException: class org.batfish.datamodel.GeneratedRoute$Builder cannot be cast to class org.batfish.datamodel.BgpRoute$Builder (org.batfish.datamodel.GeneratedRoute$Builder and org.batfish.datamodel.BgpRoute$Builder are in unnamed module of loader 'app')"," at org.batfish.datamodel.routing_policy.communities.SetCommunities.execute(SetCommunities.java:33)"," at org.batfish.datamodel.routing_policy.RoutingPolicy.call(RoutingPolicy.java:117)"," at org.batfish.datamodel.routing_policy.expr.CallExpr.evaluate(CallExpr.java:66)"," at org.batfish.datamodel.routing_policy.statement.If.execute(If.java:107)"," at org.batfish.datamodel.routing_policy.RoutingPolicy.call(RoutingPolicy.java:117)"," at org.batfish.datamodel.routing_policy.RoutingPolicy.process(RoutingPolicy.java:244)"," at org.batfish.datamodel.routing_policy.RoutingPolicy.process(RoutingPolicy.java:199)"," at org.batfish.dataplane.protocols.GeneratedRouteHelper.activateGeneratedRoute(GeneratedRouteHelper.java:47)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.processNeighborSpecificGeneratedRoute(BgpRoutingProcess.java:960)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.lambda$getOutgoingRoutesForEdge$24(BgpRoutingProcess.java:867)"," at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)"," at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)"," at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)"," at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)"," at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)"," at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)"," at java.base/java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:723)"," at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.pullV4UnicastMessages(BgpRoutingProcess.java:655)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.processBgpV4UnicastMessages(BgpRoutingProcess.java:602)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.processBgpMessages(BgpRoutingProcess.java:492)"," at org.batfish.dataplane.ibdp.BgpRoutingProcess.executeIteration(BgpRoutingProcess.java:412)"," at org.batfish.dataplane.ibdp.VirtualRouter.bgpIteration(VirtualRouter.java:1323)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.lambda$null$15(IncrementalBdpEngine.java:437)"," at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.lambda$computeIterationOfBgpRoutes$16(IncrementalBdpEngine.java:437)"," at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)"," at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)"," at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)"," at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)"," at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)"," at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)"," at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)"," at java.base/java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:1957)"," at java.base/java.util.concurrent.ForkJoinTask.tryExternalHelp(ForkJoinTask.java:378)"," at java.base/java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:323)"," at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:412)"," at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)"," at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)"," at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)"," at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)"," at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)"," at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeIterationOfBgpRoutes(IncrementalBdpEngine.java:434)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeDependentRoutesIteration(IncrementalBdpEngine.java:418)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeNonMonotonicPortionOfDataPlane(IncrementalBdpEngine.java:693)"," at org.batfish.dataplane.ibdp.IncrementalBdpEngine.computeDataPlane(IncrementalBdpEngine.java:240)"," at org.batfish.dataplane.ibdp.IncrementalDataPlanePlugin.computeDataPlane(IncrementalDataPlanePlugin.java:46)"," at org.batfish.main.Batfish.computeDataPlane(Batfish.java:806)"," at org.batfish.main.Batfish.run(Batfish.java:2197)"," at org.batfish.main.Driver.lambda$runBatfish$0(Driver.java:312)"," ... 1 more",""]}],"status":"FAILURE","summary":{"numFailed":0,"numPassed":0,"numResults":0}}

[arifogel]

@kmjmartin it looks like we are crashing trying to apply communities to a route that is generated for a specific BGP neighbor.
Because this is a vendor-neutral part of our code, it would be helpful if I could narrow down exactly which feature on which vendor we are not supporting correctly.

Are you using BGP on:

• CIsco IOS
• Cisco NX-OS
• Cisco IOS-XR
• Juniper

If Cisco, are you using any of the following in BGP:
aggregate-address with a route-map?
[neighbor ...] default-originate with a route-map?

Can you provide config snippets for these commands and the referenced route-maps? Include any related attribute-map snippets if relevant.

You can anonymize your configs with netconan if desired.

Also feel free to reach out to me (Ari Fogel) on our slack workspace.

[kmjmartin]

Hi Ari,

I've narrowed it down to occurring when the following 2 routers are in the snapshot. these are Cisco nexus routers. It doesn't look like they are using an aggregate address.

This is the config with some of the variables changed:

R1

neighbor 172.20.0.2
  bfd
  remote-as 65602
  description Peer with R2
  update-source Vlan600
  ebgp-multihop 2
  address-family ipv4 unicast
    send-community
    send-community extended
    route-map R1-to-R2 out
    default-originate route-map set_default_route_community
    next-hop-self
    soft-reconfiguration inbound always

route-map R1-to-R2 permit 10

route-map set_default_route_community permit 10
	set community 12345:1

R2

neighbor 172.20.0.1
  bfd
  remote-as 65601
  description Peer with R1
  update-source Vlan600
  address-family ipv4 unicast
    send-community
    send-community extended
    route-map R2-to-R1 out
    next-hop-self
    soft-reconfiguration inbound always
	
route-map R2-to-R1 permit 10

[dhalperi]

Thanks @kmjmartin - great report! We'll take a look.

[arifogel]

@kmjmartin The fix has been merged, and is available in this allinone test container:
test-848
You can fetch it from dockerhub via: docker pull batfish/allinone:test-848 then follow the standard instructions to get started.

Make sure to replace any references to batfish/allinone with batfish/allinone:test-848 as necessary in the instructions.