Demo stack launch order

[sharpyfox]

Hello guys. Seems that if I launch demo stack as described in wiki in this order:

1. ./build/x86_64/bin/mock_exchange_runner
2. ./build/x86_64/bin/launcher --node localhost --script ./launch.sh sample.launch.json

router failed with exception:

connecting to masterBanker/zeromq
connected to tcp://127.0.0.1:22000
zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists

----------------- Exception thrown ------------------------
type:   ML::Exception
pid:    482; tid: 482
what:   Zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists
stack:
00: 0x0x7f949bcb5e07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x7f949bcb5e07]
01: 0x0x7f949afb51d3 at Datacratic::ZookeeperConnection::checkRes(int, int&, char const*, char const*) + 0xe3 in build/x86_64/bin/libservices.so + 0xa81d3build/x86_64/bin/libservices.so(_ZN10Datacratic19ZookeeperConnection8checkResEiRiPKcS3_+0xe3) [0x7f949afb51d3]
02: 0x0x7f949afb5ef9 at Datacratic::ZookeeperConnection::createNode(std::string const&, std::string const&, bool, bool, bool, bool) + 0x439 in build/x86_64/bin/libservices.so + 0xa8ef9build/x86_64/bin/libservices.so(_ZN10Datacratic19ZookeeperConnection10createNodeERKSsS2_bbbb+0x439) [0x7f949afb5ef9]
03: 0x0x7f949af978b5 at Datacratic::ZookeeperConfigurationService::setUnique(std::string const&, Json::Value const&) + 0x285 in build/x86_64/bin/libservices.so + 0x8a8b5build/x86_64/bin/libservices.so(_ZN10Datacratic29ZookeeperConfigurationService9setUniqueERKSsRKN4Json5ValueE+0x285) [0x7f949af978b5]
04: 0x0x7f949af86a2d at Datacratic::ServiceBase::registerServiceProvider(std::string const&, std::vector<std::string, std::allocator<std::string> > const&) + 0x17d in build/x86_64/bin/libservices.so + 0x79a2dbuild/x86_64/bin/libservices.so(_ZN10Datacratic11ServiceBase23registerServiceProviderERKSsRKSt6vectorISsSaISsEE+0x17d) [0x7f949af86a2d]
05: 0x0x7f9498949ef2 at RTBKIT::Router::init() + 0xd2 in build/x86_64/bin/librtb_router.so + 0x5bef2build/x86_64/bin/librtb_router.so(_ZN6RTBKIT6Router4initEv+0xd2) [0x7f9498949ef2]
06: 0x0x7f949b460dda at RTBKIT::RouterRunner::init() + 0x43a in build/x86_64/bin/librouter_runner.so + 0x17ddabuild/x86_64/bin/librouter_runner.so(_ZN6RTBKIT12RouterRunner4initEv+0x43a) [0x7f949b460dda]
07: 0x0x40e05a at main + 0xaa in build/x86_64/bin/router_ex + 0xe05abuild/x86_64/bin/router_ex(main+0xaa) [0x40e05a]
08: 0x0x7f949a3ea30d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2130d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f949a3ea30d]
09: 0x0x40e549 in build/x86_64/bin/router_ex + 0xe549build/x86_64/bin/router_ex() [0x40e549]

terminate called after throwing an instance of 'ML::Exception'
  what():  Zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists

Though if I launch in another order:

1. ./build/x86_64/bin/launcher --node localhost --script ./launch.sh sample.launch.json
2. ./build/x86_64/bin/mock_exchange_runner

the launch is successful. I do not think that this is a serious problem, but still unpleasant

[EricRobert]

Hi,

This exception is thrown when the service tries to register itself in
ZooKeeper and someone else is already registered. ZooKeeper will keep its
'ephemeral node' around for ~5 seconds after the connection to the service
dies. So, this is probably leftovers from the router of the stack that was
just killed.

We probably should handle this exception more gracefully.

But anyhow, this should not be related. I tried to reproduce the ordering
issue between the mock exchange and the stack itself but I wasn't able to
reproduce.

Cheers,

Éric

On Tue, Apr 23, 2013 at 1:30 AM, sharpyfox notifications@github.com wrote:

Hello guys. Seems that if I launch demo stack as described in wiki in this
order:

1. ./build/x86_64/bin/mock_exchange_runner
2. ./build/x86_64/bin/launcher --node localhost --script ./launch.sh
   sample.launch.json

router failed with exception:

connecting to masterBanker/zeromq
connected to tcp://127.0.0.1:22000
zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists

----------------- Exception thrown ------------------------
type: ML::Exception
pid: 482; tid: 482
what: Zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists
stack:
00: 0x0x7f949bcb5e07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(cxa_throw+0x27) [0x7f949bcb5e07]
01: 0x0x7f949afb51d3 at Datacratic::ZookeeperConnection::checkRes(int, int&, char const, char const) + 0xe3 in build/x86_64/bin/libservices.so + 0xa81d3build/x86_64/bin/libservices.so(ZN10Datacratic19ZookeeperConnection8checkResEiRiPKcS3+0xe3) [0x7f949afb51d3]
02: 0x0x7f949afb5ef9 at Datacratic::ZookeeperConnection::createNode(std::string const&, std::string const&, bool, bool, bool, bool) + 0x439 in build/x86_64/bin/libservices.so + 0xa8ef9build/x86_64/bin/libservices.so(_ZN10Datacratic19ZookeeperConnection10createNodeERKSsS2_bbbb+0x439) [0x7f949afb5ef9]
03: 0x0x7f949af978b5 at Datacratic::ZookeeperConfigurationService::setUnique(std::string const&, Json::Value const&) + 0x285 in build/x86_64/bin/libservices.so + 0x8a8b5build/x86_64/bin/libservices.so(_ZN10Datacratic29ZookeeperConfigurationService9setUniqueERKSsRKN4Json5ValueE+0x285) [0x7f949af978b5]
04: 0x0x7f949af86a2d at Datacratic::ServiceBase::registerServiceProvider(std::string const&, std::vector<std::string, std::allocatorstd::string > const&) + 0x17d in build/x86_64/bin/libservices.so + 0x79a2dbuild/x86_64/bin/libservices.so(_ZN10Datacratic11ServiceBase23registerServiceProviderERKSsRKSt6vectorISsSaISsEE+0x17d) [0x7f949af86a2d]
05: 0x0x7f9498949ef2 at RTBKIT::Router::init() + 0xd2 in build/x86_64/bin/librtb_router.so + 0x5bef2build/x86_64/bin/librtb_router.so(_ZN6RTBKIT6Router4initEv+0xd2) [0x7f9498949ef2]
06: 0x0x7f949b460dda at RTBKIT::RouterRunner::init() + 0x43a in build/x86_64/bin/librouter_runner.so + 0x17ddabuild/x86_64/bin/librouter_runner.so(_ZN6RTBKIT12RouterRunner4initEv+0x43a) [0x7f949b460dda]
07: 0x0x40e05a at main + 0xaa in build/x86_64/bin/router_ex + 0xe05abuild/x86_64/bin/router_ex(main+0xaa) [0x40e05a]
08: 0x0x7f949a3ea30d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2130d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f949a3ea30d]
09: 0x0x40e549 in build/x86_64/bin/router_ex + 0xe549build/x86_64/bin/router_ex() [0x40e549]

terminate called after throwing an instance of 'ML::Exception'
what(): Zookeeper error on zoo_create, path /rtb-test/serviceClass/rtbRequestRouter/router: node exists

Though if I launch in another order:

1. ./build/x86_64/bin/launcher --node localhost --script ./launch.sh
   sample.launch.json
2. ./build/x86_64/bin/mock_exchange_runner

the launch is successful. I do not think that this is a serious problem,
but still unpleasant

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/23
.

[sharpyfox]

Yes, I recheck logfiles. Seems that first generated exception looks like:

connecting to masterBanker/zeromq
connected to tcp://127.0.0.1:22000
connecting to monitor/zeromq
connected to tcp://127.0.0.1:24000
connecting to monitor/zeromq
connected to tcp://127.0.0.1:24000
augmentor frequency-cap-ex has connected
connecting to agentConfigurationService/listen
connected to tcp://127.0.0.1:19000

----------------- Exception thrown ------------------------
type:   ML::Exception
pid:    25201; tid: 25201
what:   couldn't bind to any port
stack:
00: 0x0x7fc3dd19de07 at __cxa_throw + 0x27 in build/x86_64/bin/libexception_hook.so + 0xe07build/x86_64/bin/libexception_hook.so(__cxa_throw+0x27) [0x7fc3dd19de07]
01: 0x0x7fc3dc460ad2 at Datacratic::AcceptorT<Datacratic::SocketTransport>::listen(Datacratic::PortRange const&, std::string const&, Datacratic::PassiveEndpoint*, bool, int) + 0x1b2 in build/x86_64/bin/libservices.so + 0x6bad2build/x86_64/bin/libservices.so(_ZN10Datacratic9AcceptorTINS_15SocketTransportEE6listenERKNS_9PortRangeERKSsPNS_15PassiveEndpointEbi+0x1b2) [0x7fc3dc460ad2]
02: 0x0x7fc3dc46085b at Datacratic::PassiveEndpoint::init(Datacratic::PortRange const&, std::string const&, int, bool, bool, int) + 0x4b in build/x86_64/bin/libservices.so + 0x6b85bbuild/x86_64/bin/libservices.so(_ZN10Datacratic15PassiveEndpoint4initERKNS_9PortRangeERKSsibbi+0x4b) [0x7fc3dc46085b]
03: 0x0x40f3f0 at RTBKIT::MockExchangeConnector::start() + 0x60 in build/x86_64/bin/router_ex + 0xf3f0build/x86_64/bin/router_ex(_ZN6RTBKIT21MockExchangeConnector5startEv+0x60) [0x40f3f0]
04: 0x0x7fc3d9e34867 at RTBKIT::Router::startExchange(std::string const&, Json::Value const&) + 0x37 in build/x86_64/bin/librtb_router.so + 0x5e867build/x86_64/bin/librtb_router.so(_ZN6RTBKIT6Router13startExchangeERKSsRKN4Json5ValueE+0x37) [0x7fc3d9e34867]
05: 0x0x7fc3d9e34900 at RTBKIT::Router::startExchange(Json::Value const&) + 0x40 in build/x86_64/bin/librtb_router.so + 0x5e900build/x86_64/bin/librtb_router.so(_ZN6RTBKIT6Router13startExchangeERKN4Json5ValueE+0x40) [0x7fc3d9e34900]
06: 0x0x7fc3dc9487b8 at RTBKIT::RouterRunner::start() + 0xb8 in build/x86_64/bin/librouter_runner.so + 0x177b8build/x86_64/bin/librouter_runner.so(_ZN6RTBKIT12RouterRunner5startEv+0xb8) [0x7fc3dc9487b8]
07: 0x0x40e066 at main + 0xb6 in build/x86_64/bin/router_ex + 0xe066build/x86_64/bin/router_ex(main+0xb6) [0x40e066]
08: 0x0x7fc3db8d230d at __libc_start_main + 0xed in /lib/x86_64-linux-gnu/libc.so.6 + 0x2130d/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fc3db8d230d]
09: 0x0x40e549 in build/x86_64/bin/router_ex + 0xe549build/x86_64/bin/router_ex() [0x40e549]

terminate called after throwing an instance of 'ML::Exception'
  what():  couldn't bind to any port

and only after this crash router fail with zookeper exception.

[EricRobert]

That's interesting...

I pushed a very small update so that the exception actually prints the port range it tries to bid to. If you could run that and check that this port is actually not used on your machine, that would be helpful.

Thanks!

Éric

[sharpyfox]

I've modified your code a bit, and receive this message:

what:   couldn't bind to any port in range [12339, 12340]

[sharpyfox]

Hello Eric, do you have any news on this issue?

[sharpyfox]

I'll do some research and saw that connect in rtbkit/common/testing/exchange_source.cc always return 0(success) in my case. Even if demo stack is down. However errno = 111(connection refused) after connect call. I've modified code a bit (add 0 == errno check) and it works properly for me now. Should I create pull request for this?

[EricRobert]

Hi,

Yes, please do. I was just investigating something else and it seems that a
recent change to the bid source broke something.
In my case, it works for some time but it then stops.

Thanks!

Éric

On Wed, Jun 5, 2013 at 2:36 AM, Nikita Vasiliev notifications@github.comwrote:

I'll do some research and saw that connect in
rtbkit/common/testing/exchange_source.cc always return 0(success) in my
case. Even if demo stack is down. However errno = 111(connection refused)
after connect call. I've modified code a bit (add 0 == errno check) and
it works properly for me now. Should I create pull request for this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/23#issuecomment-18958041
.

[RAttab]

Seems like everything is resolved here so I'm closing this issue.