Autosubmit Interval failling...

[pedrohenriquerls]

Hi, I have a service to send info to Librato in interval of each 5 minutes, yesterday I become to have the error {"errors":{"params":{"measure_time":["is too far in the past"]}},"request_time":1475920599}", everytime a long of one entire day.
My doubt is... How will the Librato gem send a value too old if my autosubmit_interval is equal to 5 minutes?

This is my service:

require 'librato/metrics'
require 'singleton'
class LibratoService
  include Singleton

  AUTOSUBMIT_COUNT=5
  AUTOSUBMIT_INTERVAL=5.minutes

  def initialize
    Librato::Metrics.authenticate(ENV['LIBRATO_EMAIL'], ENV['LIBRATO_API_KEY'])
    @queue = Librato::Metrics::Queue.new(autosubmit_interval: AUTOSUBMIT_INTERVAL)
    @aggregator = Librato::Metrics::Aggregator.new(autosubmit_interval: AUTOSUBMIT_INTERVAL)
  end

  def aggregator_watcher(metric, value)
    @aggregator.add("#{prefix_metric_name(metric)}" => value)
  end

  def latency_watcher(metric)
    @queue.time prefix_metric_name(metric) do
      yield
    end
  end

  def send_gauge(metric, value)
    Librato::Metrics.submit "#{prefix_metric_name(metric)}" => value
  end

  def send_metrics
    @queue.submit
    @aggregator.submit
  rescue Librato::Metrics::CredentialsMissing => e
    LoggingService.log(e.message)
  end

  private

  def prefix_metric_name(metric_name)
    "Notification.#{metric_name}"
  end
end

[niklibrato]

The "measure_time":["is too far in the past"] error is triggered if the time stamp submitted with the measurement is more than 2h behind wall time. Could it be that the clock on the server that is submitting the metrics has drifted?

[pedrohenriquerls]

Thanks for the quick reply.
this was my first thought, but there're no changes on the clock server and this service was working fine a couple months 😕, you have another guess?

[nextmat]

@pedrohenriquerls - just wanted to check back and see if you were able to get this resolved? That message actually gets parsed straight from the API response so the only way you should see it is if the specified time for the measurement is significantly in the past.

One possibility that occurs to me is that you could be adding metrics sporadically enough that your autosubmit is actually sending less frequently than every 5 mins. From the related docs:

...are driven by the addition of measurements. If you are adding measurements irregularly (less than once per second), time-based submission may lag past your specified interval until the next measurement is added.

We have had some people confused by this in the past. The autosubmit is only checked when you add a metric so if you are adding them rarely you may be submitting at longer windows than you think.

If this is the issue you can roll your own looped autosubmitter easily using something like this example or if you are wanting to report regularly via a background thread you may want to us something like librato-rack instead.

[nextmat]

@pedrohenriquerls - going to go ahead and close this, feel free to follow up with support@librato.com or here if you have further issues.