The ext/debian/changelog file is quite outdated and gives the appearance of a very old (outdated) package. The rake task for the debian package builds generates the changelog on the fly, so the file can be safely removed. Signed-off-by: Michael Stahnke <email@example.com>
Previously the rake taks to build a Debian package would not contian the source files. Commit changes behvaior to build source files as well. Patch from: Max Bowsher Signed-off-by: Michael Stahnke <firstname.lastname@example.org>
Allow the Message object to be of type :direct_request which would force direct requests Change identity filters to be and and not or since that makes more sense for identities. But also if we're sending messages with many identities attached we'd only want to match if ours is anywhere in the list, an and based selection would always fail. If identities are provided on the CLI then we use those and avoid discovery. The RPC::Client#discover function can now be passed a list of hosts in either string, array or JSON formats. It will extract from this data discovery data and use that instead of the old discovery methods. In this use case it will force direct_addressing mode. The rpc application can now take JSON on STDIN and will use the above capabilities to fill in discovery data
As messages are now possibly sent to queues and can hang around the middleware for a very long time we need to support a TTL. Nodes that go off-line between discovery and request being sent may come back a month later and find these old messages sitting there and then take the actions which would not be desirable. Requests will now get a default TTL of 60 seconds and during validation of a message any messages older than this will not validate. The various security plugins will accept a TTL when encoding the messages and default to 60 if not given, similarly the Message object will default to 60 TTL. As a side effect security plugins now need a TTL when encoding requests which means we now need to upgrade all security plugins in use when going to a version running this version. Future tickets will allow the TTL to be set per message.
Improve the documentation for the registration_collective configuration by adding hints about what versions support this and how older ones behave
The default classesfile location was set to a location that Puppet never defaulted to. It was missing the state part of the path, and the more modern classesfile location in puppet also has lib in the path. This is now the default.
The authorized_by helper creates a method called authorization_hook and loads up a class from disk that provides the implimentation as a plugin. We used to only create 1 instance of a agent so this only happened once but recently we started creating new instances of the agent on each request which means this steps gets done many times. This change avoids creating the hook if it's already defined thus avoiding the undesired disk loads
DDL files can declare a input data item can be boolean which means we would like to validate that on the agent side. The validators had no boolean validation so this adds one. Also do a small bit of test maintenance to avoid spurious failures
Tweak the justification of multi line text results so it all lines up nicely
…l execution in environment. We now raise exceptions when fact filter parsing fails, the -W helper relied on it returning false. This adds support for the new behavior to -W
…l execution in environment. If users supplied fact filters on the CLI or in ruby code that did not match our expected valid fact formats the effect would be as if the filter was never specified. This would result in requests being run on the wrong machines. We now raise an exception rather than just returning false
During the work in #8181 a new filter type was introduced but a few integration points were overlooked. The result of these were that for some kinds of requests no filter were applied. This improves the Util#empty_filter method to return an empty compound filter and also improves various aspects of the RPC::Client class to be aware of the compound filter.
Add a small filter language that allows rich expressions to be used when doing discovery. A simple parser/scanner was included in MCollective::Matcher and it's available on all SimpleRPC clients with the -S option and on any MCollective::RPC::Client using the #compound_filter method.
The RC scripts were setting the RUBYLIB environment variable, this should not be needed as the operatinsystem should do the right thing.
Add the ability for the printrpc helper to return JSON data. Any application that is using the printrpc helper to display data will now be able to just add -j to the command line and return JSON instead. If STDOUT is a tty the JSON will be pretty formatted else it will just be a big blob of machine friendly JSON. The basic use case is to eventually allow piping RPC results from one query into the following query - perhaps with some grep like filtering - thus avoiding discovery and allowing totally arbitrary actions on arbitrary sets of hosts. This also adjusts the progress bar and the stats to only display if STDOUT is a tty.
This adds a new direct request method to the Message object that will communicate with nodes directly. With this feature enabled requests that goes out to a number of nodes below a certain threshold (10 by default) will use the new point to point mode while the rest will use broadcast. This should be transparent to the end user. But is disabled by default The idea with this is to enable communication with off-line machines but also to optimize the communication where you need to send a message to only 10 or so nodes and creating 10 messages would be more efficient than sending a broadcast message to all of 1000s of nodes, The other area where this is important is in the web space where users can easily create arbitrary hosts lists by clicking on nodes. In this scenario the grouping based on facts, classes etc does not apply well. Having a direct mode means you could supply arbitrary hosts lists. Ultimately this is the ground work for making discovery pluggable and optional. In the case of the web scenario above when the web application provides a list of hostnames to communicate with the mcollective infrastructure will not do a discovery but simply go and communicate with those nodes. This specific commit begins making the various internals aware of this new mode, it adds 2 configuration options but leaves this mode disabled by default. The stomp connector will create queues per node to enable this, doing it this way is pretty inefficient and wont work at scale a new ActiveMQ connector will be created that use a much better but ActiveMQ specific approach to solving the queues per server problem. Followup work needs to be done around request TTLs to avoid down machines from doing lots of unintended work.
During the work for #7619 the Client#receive were changed to return Message objects but the #discovered_req method that is mostly unused were not made aware of this change. This commit rectifies that.
When using the pooled configuration syntax you now automatically get detailed logging at info level for connection events on the Stomp connetion. This is based on a feature in Stomp 1.1.9 where you can have a class that responds to some predictable method names to create call back based logging. The stomp plugin constains a class M::C::Stomp::EventLogger that has these methods and does the logging using the normal logging framework.
add in jetty and the environment variables so the web console is active
Decrease log level on missing DDL file messages in the agent from warning to debug as at present there's no real reason for them to exist on the servers anyway.
Add a new config option direct_addressing that will cause the main runner to request :directed subscriptions from the middleware plugins. This setting is off by default since the default stomp plugin really isn't going to be able to do really intelligent stuff with this and on big networks it will have big impacts on middleware. The proposed ActiveMQ specific plugin will be able to do this in a way that does not have such a huge impact on the middleware. For Stomp we've added a new config option queueprefix that compliments topicprefix and the plugin will simply subscribe to a queue that has a MD5 of the configured identity. We're using the md5 as thats the safest way to ensure there's no illegal characters in the resulting queue names. The intention with this ability is so that we can address individual nodes directly without broadcasting. The use cases for this are - among others: - Web apps where people will be clicking arbitrary hosts, the library will fall back to direct mode in that case - For calls to M::RPC::Client#custom_request where the list of expected replies are some arbitrary low number like 20. In a big network talking to those hosts directly will have a lower impact than the broadcast mode - Add the potential to do batched requests. With the broadcast mode it's a bit all or nothing unless restrictive filters are generated. By combining the broadcast for discovery with directed requests in groups of n hosts we can slowly go through a network rolling out change in managable batches. - This will enable truely pluggable discovery. External resources like BPM systems can supply a list of hosts they wish to talk to and we can do that easily. - Queued requests for off-line machines None of these abilities are in the libraries now, this is just the enabling architecture.
…nnectors Previously the structure of the middleware was baked into the mcollective core we had assumptions about how the middleware looked and this made it hard for connector plugins to be effective. Connector plugins had to abide by the internal assumptions rather than have the freedom to impliment middleware however they wanted. We deliberately threw away message headers as received from the middleware as they were considered too middleware specific. This was a mistake as effective use of the middleware relies on these. We now have a MC::Message object that encapsulates all the properties of a mcollective request or reply but in a way that does not assume a structure of the middleware. The connector plugin now gets to make all decisions about the middleware structure. All creation of topics, targets, subscriptions etc gets delegated to the connector Additionally the Message object now takes care of validation, encoding, decoding and base64 processing of messages as these functions only apply to message objects. This simplified a lot of code especially in the main runner loop and improve out ability to extend mcollective for the work related to #7226 The entire server and client logic has been reworked around this new object and lots of dead code in the Util class was removed. In the case where a Message object is actually a reply to an earlier Message the earlier message is stored and the connector plugin will have access to the full request that initiated a reply. The connector will thus have access to all the headers received and can make routing decision based on those. The core has no use for message headers, this is purely to support the connectors. This will make doing #7246 and #6938 much eaier This change also breaks backward compatability with early 1.1.x and earlier, this is acceptable as the current production release is 1.2.x and we maintain compatability with that. The message protocol now has proper headers for the agent and collective a message was sent to so the parsing of :msgtarget was retired. So this header was removed and old servers just wont understand these messages and we wont understand old clients.