Skip to content


first stab at a graphite listener #293

wants to merge 6 commits into from

8 participants


this is a RFC. it has some comments, questions and TODO's in the code.

most notably needs some work

@pauldix pauldix commented on an outdated diff
((137 lines not shown))
+ if len(elements) != 3 {
+ continue // invalid line
+ }
+ val, err := strconv.ParseFloat(elements[1], 64)
+ if err != nil {
+ continue // invalid line
+ }
+ timestamp, err := strconv.ParseUint(elements[2], 10, 32)
+ if err != nil {
+ continue // invalid line
+ }
+ // this doesn't work yet. should i use the types in protocol/types.go ?
+ // they don't seem to match with what's in protocol.pb.go
+ // and the latter seems a little lowlevel and has attributes that don't apply in this context
+ // also, not sure if i still have to reserve a column for time or sequence number, or whether
+ // that will be handled automatically for me
@pauldix InfluxData member
pauldix added a note

You don't have to set timestamp if you call to WriteSeriesData as the points come in, it'll be assigned automatically. Sequence number will always get assigned automatically.

You should be using the objects from protocol.pb.go. Basically, create a series object, columns, and then points. Create field values for the points. Timestamp and Sequence number should not be their own fields.

This might give you a better idea:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@Dieterbe Dieterbe referenced this pull request in graphite-ng/graphite-api-experiment

merging projects, uniting contributors (if it makes sense) #7


pushed a bunch of fixes, but now my getAuth() doesn't work because (at startup) len(names) == 0.
how should i fix this? shouldn't always at least 1 admin user exist?

// getAuth assures that the user property is a user with access to the graphite database
func (self *Server) getAuth() {
    // just use any (the first) of the list of admins.
    // can we assume there's always at least 1 ?
    names := self.clusterConfig.GetClusterAdmins()
    self.user = self.clusterConfig.GetClusterAdmin(names[0])
InfluxData member

It could be empty if the Raft server hasn't started up yet. I would pull the setting of the user credentials outside of the New method for the listener and instead set them when you call ListenAndServe. And make that happen after everything else has been initialized.


I think this is ready for review. I tested it by feeding it data using the diamond agent and that seems to work fine.
2 notes though:

  • still need to implement check if db exists and create if needed (paul you said you guys would do this IIRC)
  • I did see some errors in the output wrt raft/shard logs, which is seemingly unrelated to the graphite listener. see should i open a new ticket for this?
@pauldix pauldix commented on the diff
((153 lines not shown))
+ } else {
+ values = append(values, &protocol.FieldValue{DoubleValue: &val})
+ }
+ ts := int64(timestamp * 1000000)
+ sn := uint64(1) // use same SN makes sure that we'll only keep the latest value for a given metric_id-timestamp pair
+ point := &protocol.Point{
+ Timestamp: &ts,
+ Values: values,
+ SequenceNumber: &sn,
+ }
+ series := &protocol.Series{
+ Name: &elements[0],
+ Fields: []string{"value"},
+ Points: []*protocol.Point{point},
+ }
+ // little inefficient for now, later we might want to add multiple series in 1 writePoints request
@pauldix InfluxData member
pauldix added a note

Agreed. #310 will add that functionality to requests, which will make it possible to update the interface on the coordinator. Will definitely make things more efficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
InfluxData member

This looks good to me. We can help with the database validation. I'd like @jvshahid to take a look too.

For the Raft log thing, log an issue with as much info as you can and we'll see if we can repro. Thanks!

@Dieterbe Dieterbe referenced this pull request

raft/shard log errors #311

@zorkian zorkian commented on the diff
((62 lines not shown))
+func (self *Server) ListenAndServe() {
+ self.getAuth()
+ var err error
+ if self.listen_addr != "" {
+ self.conn, err = net.Listen("tcp", self.listen_addr)
+ if err != nil {
+ log.Error("GraphiteServer: Listen: ", err)
+ return
+ }
+ }
+ self.Serve(self.conn)
+func (self *Server) Serve(listener net.Listener) {
+ // not really sure of the use of this shutdown channel,
zorkian added a note

I think the shutdown here doesn't do much; or at least, not what you hope.

It works in the HTTP API because the underlying net/http module's Serve() method has some intelligent handling of Accept() failures:

In short it determines if it's a temporary or permanent failure, and it aborts the accept loop on permanent failures. That allows the Serve() method to exit, which ends up triggering the defer to send the true on the shutdown channel.

I haven't fully thought through the implications of shutdown in InfluxDB (just started looking at the code today) so I don't know what the right solution in API modules like this is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

(Thank you for doing this; I've used it as a base for my own OpenTSDB ingester: zorkian@b57b611 ... not done yet, but it's a start and you should recognize the structure!)

@jvshahid jvshahid added this to the 0.5.0 milestone
@jvshahid jvshahid was assigned by Dieterbe
@jvshahid jvshahid modified the milestone: 0.6.0, 0.5.0

it is nice to see some other listeners but wouldn't be better if there was a nicer way to have each plugins have its own config without having it merged like that in the rest of the config file ?

  name = "http_json"
  port = 8086

  name = "udp_whatever"
  port = 6666

  name = "http_graphite"
  port = 67888

instead of the current model which seems to be:

  port = 8086

  port = 67888

This is just an idea but it would help cleaning up the config file and show clearly what is an input plugin and allow users to quickly turn them on and off by commenting the whole block.

another idea:

    port = 8086

    port = 6666
    other_param = 42

    port = 67888

The specific syntax does not matter that much to me, what matters is that the plugins are clearly identified in the config and not only by comments.

InfluxData member

I agree, the last example you posted looks best to me.


I am not familiar enough withe codebase but could there be a way to have plugins code cleanly split from the core (and to an extend maybe not compiled at all when not needed) ?

Here is an example of what I meant (again that's not the way things are laid out but more the idea behind it):



I am sure that problem will arise sooner than later as more plugins are added especially if they need specific libraries not needed by the influxdb core, another factor is that once some plugins are out there the "architecture" won't be as easy to change as it could be now.

One of the plugins I want to give a try at is json over udp, when the server writing to influxdb is on the same machine I am sure it could provide a nice boost over http and completely remove its overhead.


Any plans to make this ingest the Graphite pickle protocol as well?

Here would be my use case:

I currently have a relay in every data center that forwards data from every server to a data center local graphite instance and a centrally hosted instance as well. So I have one point, where I have all data center specific metrics, but I also have a global view. Problem is that the global instance is getting under water more and more.
What I plan to do is relay data to a global InfluxDB cluster as well and evaluate if it is something I would bet on for full production use (it does sound very promising, but I need some test data).

So what do you think? Maybe there are there better options for me?


I'm not a big fan of the pickle protocol. seems like it got added to some of the python daemons because somebody thought it would be cool but IMHO in reality it doesn't help with anything, is hard to debug, and breaks compatibility between different services.. tools that only support pickle protocol should support the plaintext protocol and in my view everything should just use the plaintext protocol. (or if you want to optimize, something truly cross-language such as msgpack, but that's a whole other story)


I am so with you!

But I think my current setup is actually a common one and it might be something to consider to help people ease into InfluxDB.

I am not sure if there is something similar like the carbon relay for Influx. For me, this is pretty vital, because I do not want every server to be able to directly talk with InfluxDB and do not want to open up a port to the outside world for it. A central relay that could handle this would be cool. But I guess that does not fit into this PR :)

@jvshahid jvshahid added a commit that closed this pull request
@jvshahid jvshahid Close #293. Merge branch 'pr-293' a11a11e
@jvshahid jvshahid closed this in a11a11e

Thanks @Dieterbe. The pull request is merged in. I made some changes and added a test.


cool. note that there's one thing still left todo

~/w/e/influxdb ❯❯❯ ack TODO src/api/graphite/api.go
// TODO: check that database exists and create it if not

(paul said this would be easy for you guys to add and i didn't have to worry about it :))


btw @Frusty here's a relay that talks the plaintext protocol. I'll be using it to send metrics to both carbon and influx.

@jvshahid jvshahid modified the milestone: 0.5.0, 0.6.0
@Dieterbe Dieterbe deleted the Dieterbe:graphite-listener branch

@Frusty the pickle protocol is too complex to implement reasonably in another language because the pickle itself is tightly tied to the implementation of python.

You're much better off transporting with the line protocol for now.

@mindreframer mindreframer added a commit to mindreframer/golang-devops-stuff that referenced this pull request
@mindreframer mindreframer Squashed 'src/' changes from 7d3b31b..63…

63cc69f Add a test for shard spaces regex
d3e27e8 Fix a typo
dcbadb9 We should just keep track of the time object instead of the seconds
7a297b3 Close #803. Fix #767
8607a60 Make sure we remove the shard metadata whenever it expires
adfbdb0 Add a test for the shard expiration
22d02b7 Update
69ba4ef Merge pull request #426 from influxdb/fix-426
680e6d9 Always fill empty groups if the start time is specified.
e74b941 add rc5 to the changelog
031f4ad Remove an unused statement
b16e090 Add a standard response markdown file
dd116bd Don't read the entire body in memory
9750840 Move load database config to API
eaaa322 Update
72fccda Don't emit non existent fields when joining
3ebf536 wait for servers to sync
22cb673 Fix the build on go1.3
e8f7503 use camel casing instead of underscore
67f9869 revert previous commit
99ac385 speed up the release
d18359a update the changelog
856a94c graphite ingest write data in batches to coordinator
3284662 Fix an integration test that has changed in 91078c0
e349e22 Fix a unit test that has changed in 91078c0
e196277 Merge branch 'pr-738'
673a122 Dry the fill() tests
c1e2853 Allow 'null' to be supplied as fill value
91078c0 Non admin database users shouldn't be able to drop series
b051ffc add a flag to force output to go to stdout
f1f575d Fix travis
c02cff2 Fix some bugs with retention policy of shard spaces
3b710fd update the changelog
afe3f96 Add the sentinel values for all db on creation
6dfc2b3 Add String() to the Field structure
588e053 Merge pull request #766 from shugo/database_conf_fix
5d4e2bb Fix the name of the config files in the integration test suite
492fe30 Use goroot to find go and gofmt
87a68be fix typo in integration/database_conf.json.
b4d524a Fix the test to use client.ShardSpace
5b9ab34 fixup! use the client library without alias
cfde05d Add New() to the client library so users can do client.New()
8728b25 use the client library without alias
f6ae65d Fix the package name of the client package
fd45c92 Remove more indirect references to the protocol package
e4c25d0 Revert "alias SerializedSeries"
5e1b7ea update the changelog
1e9e215 Do the check in one go
7b0fffe Merge remote-tracking branch 'refs/remotes/origin/pr/760'
05a9356 Merge remote-tracking branch 'refs/remotes/origin/pr/759'
13514dc more stuff in the contributing doc
8511f40 Add some links to Go related posts
07363e8 remove devtools since we don't need it anymore
ab42581 Merge pull request #758 from otoolep/better_graphite_errors
d74e71c When a server is removed, its ID should be removed from shards.
8646cb5 The attributes of a shard space should not be reverted to the defalut values when writing data into the shard space.
01d4762 Set upper limit for UDP server
4271c47 Make it crystal clear why graphite fails to start
1f180dc fix the build on centos to use recent rocksdb and statically link stdc++
744edc5 don't die if clock_gettime can't be found
4207442 Uh, fix caps on contributing...
df7e4d2 fix the client test
aeb21aa check if -lrt is needed for clock_gettime
6c5870e update the changelog
8972d73 use devtools g++ 4.7 to be able to compile rocksdb
3dfe437 lower the write buffer size
bf054b9 update the contributing guide to have a link to the CLA
7526968 alias SerializedSeries
572450c Merge pull request #743 from neurodrone/master
3fc281e Up default max-open-files
5503d54 Prevent redundant copies when pulling in queried data-points
4f0e6f1 move more docs to the docs directory
54410a2 more typos
29ad12c fix a typo.
89b6da1 Add more dependencies that are needed on Debian 6
9820ce4 Add `make format` to the doc
fba1706 Add a contributing doc
796f922 Move some files around and delete ones we don't need
febd8cd update the changelog
6669f0c disable arm release until it's fixed
100ac49 use whatever go is on the path
51f5a9e Lower the default map size to make it work on 32 bit machines
2833930 don't delete the admin directory
7151db2 when packaging use the new names
93394a0 remove an unused file
8dc5303 build the protobuf go generator if it doesn't exist
427ea50 force the output to go to the expected file name even with redirects
ba79c9b Fix the integration test suite to use the cluster type structures
c83fbce change the default storage engine to RocksDB
510b558 Update
d40a196 Use the same ShardSpace in the server and client
1d27706 Update datbase json to be idomatic
25c1097 Update
ecb42f8 Update
6285757 Use influxdb/influxdb/client instead of influxdb/influxdb-go
b3038c0 Merge branch 'master' of
f336acc Update database conf to have proper order
941bc97 delete the client license and gitignore
a915bf5 Subtree merged client code
be250db Have the server periodically clear out old shards
c84f249 fix an integration test
aac69f0 Remove a Printf
39ea797 Fix #690. Use idiomatic go project structure
e798e2b Update
9819282 Merge branch 'retention-policy'
4be5fc7 use one field instead of two in the test
9993b18 Add command line option to load a database configuration.
2c9ccd7 Return an error if there are no shards to query
d59975b Get rid of the shards slice
a497ce2 Add Shard Spaces and Retention Policies
23dc106 update the CreateShardSpace to not include db in the url
7d5d7e5 Merge pull request #13 from apg/versiontest-typo
6474f81 Compare to the reference version, not the equivalent version.
27a6e3a Merge pull request #735 from otoolep/ignore_vim_temp
5bf7479 Fix json marshal, add database to space
a6750fd Ignore vim temp files
3a4cdf8 Add fix on version check for in development versions
91d9d4a Don't link the rc versions to latest when uploading the packages
58f132d Parse shards depending on the version of InfluxDB
b4eb6b7 Add database as part of the API
c6c97ec update the changelog
5b9068a Support milliseconds duration using `ms` prefix
04ce6a2 Update drop shard space to take database name
0b6371e Writes to non replicated shards shouldn't be buffered or use the WAL
3fb936f Graphite plugin should work with payload that are whitespace delimited
0243d78 Update the changelog
191b575 Do not start the UDP input plugin if it's disabled
dfaea57 update the changelog
0a4f714 Merge pull request #689 from influxdb/fix-414-NEW-move-series-metadata-to-raft
9c12fb7 Move time series metadata to Raft
0b13c22 Change ShardSpace to SpaceName for consistency with server side
bdea460 Ugh, marshal needs a pointer, derrrr
ac2e0a6 Fix marshal of GetShardSpaces
d174bc2 Change GetShardSpacesAndShards to just GetShardSpaces
0252511 Add v2 version of getshards
d0873fb Add split to shard space
b9a98c2 Add replication factor to shard space
0bb03a5 Add comment to describe shard duration
c03dbae Add shard duration
91c6ca3 Add methods for working with shard spaces
8a10160 Merge pull request #723 from chobie/fix-669-cast-to-value_array
133be2c cast *C.value_array from *C.struct___0 #699
3fe0444 Update the change log
2b9d32c Close #716. Fix #702. don't panic when dropping non-existent shard
4046c45 Fix #12. Disable compression by default
76c5024 Close #682. Add backfill option to into clause parser
39acc29 Fix a broken test
e5de7d3 Close #684. Return the permissions when listing a db user
9508c16 Close #695. Detect duplicate field names
94819a0 remove which is unused
cb217c6 Close #10. Add udp support
76285b6 use go 1.2.1 and 1.3 on travis
e527611 Restructure the codebase to be more idiomatic
397de00 Merge pull request #687 from mdlayher/master
f3ecb1e Don't clean the makefile
4493b01 Don't sed leveldb, hyperleveldb or rocksdb
9bcba03 Merge pull request #11 from pmbauer/master
9506cbb handle http get error in GetShards
dc2cb48 Fix the udp api to use json.UseNumber()
61cbbd3 Fix typo and trim trailing whitespace in config.toml
cb7d361 Clean should clean the temp directories and all ignored files
d645933 Fix the build so hyperleveldb, rocksdb and leveldb all use snappy
53eefdd don't build the shared rocksdb library
857c7a9 Add QueryWithNumber to the api
792468d Don't compile hyperleveldb and rocksdb with snappy for now
a704896 Fix the tests to work with ints and floats properly
083c30d Close #677. Extract coordinator.Permissions
574646b Use absolute path of gofmt
2204fba Fix #672. Build hyperleveldb and rocksdb with snappy support
96d2622 Add vet target to Makefile and trivial cleanup
e586916 Close #680. Format our code base
9dac392 add a pre-commit hook
1783229 Add format and check to the makefile
5a3ea36 update the changelog
dcc1c9a Fix #676. Allow for high precision integers to be stored correctly
00f71fa ignore the admin symlink
20e7f3a Merge pull request #667 from influxdb/compression
7188032 Merge pull request #665 from dgnorton/master
286a106 Fix the tags in the Makefile
e6a1812 Fix the chunk response when compression is enabled
5b1d36f whitespace
cde446d Enable compression on all GET requests
990fbdb Enable compression when posting time series data
b114bd0 Merge remote-tracking branch 'refs/remotes/origin/pr/666'
1621876 Use our fork of gomdb
86a28be Fix #670. Redirect output to /dev/null
d9f861e Merge pull request #9 from influxdb/compression
e83482d Change default max-open-files to 1000 because the low setting is making too many people sad.
6165d81 Use cursors for batch writes
af1c7c6 Add -path option
f454946 Make compression configurable in the client
5911676 Fix delete
ef0f5eb Make start udp server log message more detailed.
16781e3 Made build tmp directory configurable via TMPDIR.
4c704f0 conditionally build hyperleveldb
8c65dbc Disable hyperleveldb and rocksdb on Mac
0fe044e Merge pull request #661 from bcrisp/master
0057fb6 fix the awk in the make file
a8b81d0 Check for the libz, libbz2 and g++-4.7 before building
1790b18 Update
f6735c8 Update
19c4dd1 update the changelog
97d8bcf Merge branch 'b-tree'
2563cc5 Make rocksdb optional, since we can't build it on travis
cfc4525 fix the directory
390eee1 Don't close the cache which could still be used
994bc20 fix the timing of the second write
a9b1bdf Add support for hyperleveldb
7ce4b94 Add support for rocksdb
18b3b4f keep track of the database type
ffbe9be Remove LevelDb from the shard and the shard data store
d790a72 Make the storage engine selectable from the config file
319e6a4 Abstract storage engines
c89f03d Close #643. Support pretty-print of JSON via HTTP API
1e8c746 Fix a typo
7fac139 Fix influxdb on 32 bit arch
2408ac8 update the changelog
39fb8d2 Merge pull request #651 from Dieterbe/patch-1
fda4194 follow symlinks
216a3eb Update
d2999d0 Update
06b8743 Update
28119db Merge pull request #646 from influxdb/fix-640-duplicate-shard-ids
00cf2d6 Fix #640. Update cluster config to track the last used shard id and increment for new ones
a678aa4 Fix delWithBody method to take an interface of io.reader
21bb706 Update the changelog
d3ffecc Fix #637. Truncate log file if the last request wasn't written properly
33a5c65 Add dropshards
84685bb Make getshards return objects
687a2d9 Add method to get shards
3215453 Protect protobuf client from simultaneous calls to reconnect()
ba94bd4 Close #636. Support `ms` as well as `m` as time precision when querying
203009a Merge pull request #626 from pmenglund/master
5e1411a extract init config file into a variable
b7fc984 Merge pull request #620 from freeformz/use_interval
9aa26e1 Use the heartbeat interval as the timeout
8bd802f Cleanup (Thanks, @freeformz)
773cf45 Fix #608. Fix merge when one of the series doesn't yield any points
0ff03ba Merge pull request #612 from freeformz/fix_benchmark_mistake
709a7a5 Merge pull request #613 from freeformz/benchmark_error_formatting
a3294e8 Print Error()
b318153 Fixes a bug I introduced with SSL for testing
63c2be7 update the changelog
5485d29 Disable the second argument of the MODE aggregation
3ecd697 Add MODE tests and fix a bug
7b19431 DRY and modify the code to be idiomatic
cc5a5e8 Close #521. Allow MODE to work on all datatypes
527517b update the changelog
a3d8b37 query isn't always a select query
70fb954 Fix some log statements
caa8481 Fix #602. Merge will fail to work across shards
7c753e6 update the changelog
7ddccac Close #605. reduce some duplication
d40ed03 No redirect to log.txt and log instead of writing to stderr
af78db4 Run the query as root
e43d07b Update
de0a938 Merge pull request #418 from influxdb/fix-418-break-large-requests
7498a36 Fix a broken test
c89a6d3 Fix the logic that breaks the repsonse
b626345 Use GetClient in the test to get a client with the right port
e9b2ce5 Use Size() instead of encoding
6e46865 Fix #418. Make sure none of the writes exceed 2M
b5bb038 Merge pull request #603 from freeformz/ssl_all_the_things
add2df5 Use SSL if we have the config for it
791215d Merge pull request #601 from freeformz/benchmark_changes
b7e104a Optional TLS and timeouts for statsServer
7d4cc6e Looks like you were implemeting a WaitGroup anwyay...
153e111 print the correct version
d36bc7e Update
fd2f6dd fix some typos
b82fc80 fix the release script to use crosstools
421fc0e simplify the code since we don't need the quit channel
07891d4 Update the changelog
45c3d68 Close #600. Fix #599. Report os, arch, and raftName once per day.
18e2e5e Fix #597. Force compaction after deleting data
396b5b7 Fix some tests broken by #579
ad85345 Close #566. Add some timing data to the logs
a047866 Fix #579. Reject writes to nonexistent databases
16deb08 Remove some unused fields and fix the test
9a6cc74 More idiomatic and DRY
90e9af0 Close #591. Add support for multiple UDP + JSON input plugins
7871bf4 err could be EOF according to the docs
5131d48 Update
83925a6 Merge pull request #544 from influxdb/fix-544-force-remove-node
100d629 update the name of the package
08fb487 Fix #559. Add a test to demonstrate that this isn't a bug
9598b61 update the changelog
54cbbd6 remove a print statement
7710049 add a test for #578
cbdebd8 Merge pull request #578 from chobie/fix-496-parentheses-value-should-support-alias
8035fd9 parentheses value should support alias #496
4e10062 Close #476. update makefile to add support for arm builds
9ccc4af no need for the reference to self
696d61d prepare for a release
2dbbbc6 update the changelog
ed5769b Close #560. Use /dev/urandom instead of /dev/random
dd26730 Fix #502. Fix a race condition in assigning id to db+series+field
ba0250c give credit to @peekeri
eaed57f don't always assume that points have timestamp and/or sequence number
bf2c1c6 Close #567. Fix #112. parser: allow select from multiple series
b96b36e change the error message in the test
775e849 update the changelog
1f7fd31 Fix #576. Don't set timestamp or sequence numbers when listing cq
67c30bc Don't call time.Now() unnecessarily
afe2352 Fix #575. Selecing single point queries don't interpret timestamp correctly
83cfec1 Close #443. Use `name` instead of `username` in /cluster_admins
0914e8e Fix #565. Make sure empty series names don't cause a panic
953137a Fix #537. Incorrect query syntax shouldn't causes internal error
a6547c5 Update
5b984c1 Fix the build instructions link and format the readme
e6c828f fix a typo in the changelog and add udp to the graphite docs
c42d8dc Fix #563. Add sample config for graphite over upd
b0f99d0 Close #561. Fix missing query string in parser errors
9d91b41 Fix #544. Add a way to force remove a node
63c9a5f add functions to get and remove servers
062de96 Fix #524. Arithmetic operator and where conditions don't play nice
bd03a57 update the year in the LICENSE file
890f291 Fix #550. Fix tests on 32-bit ARM
9a06ec4 Close #547. Add difference function
86a80f7 fix the style of the changelog
af98eca Fix #529. Don't run influxdb as root
9c8204b Fix #460. Don't start influxdb automatically after installing from pkg
b6a5a10 Fix #557. Using group by time(1y) doesn't work while time(365d) is fine
a64ee2a update the changelog
44688fc remove newlines from log statements
1369ca7 Fix #555. Fix a regression in d593348 where the snapshot format changed
379215f add some logging
5b31854 fix a log statement
39e659c update the changelog
d1b0597 Merge branch 'pr-551'
a08e63e more idiomatic go
3c8aafd modify some tests
d382ea0 Close #551. Fix #409. Add aggregate funciton top and low
a4ece2a Update
0744a29 Update
6e81a63 Merge pull request #545 from freeformz/typo
6a74cfd Merge pull request #546 from peekeri/graphite_udp_api
50d8050 graphite api: add support for sending metrics over UDP
29d6330 Fix typos
bd33adf Fix #540. Add a test to make sure regex matching with literal dot work
d593348 Fix #489. Remove replication factor from CreateDatabase command
702bf3d update the changelog
964f5b2 don't append to the original slice
b3fac38 Optimize deletes
8286ced Optimize writing to multiple series
d6050c9 remove some unused constants
998cf1e Make the write batch size configurable
c7c79bc Fix #531. Compare s1's to s2's name
645aac7 update the changelog
2e0f6c1 Fix #534. Create a new series when interpolating
2d11e22 Use the abbreviated commit sha
3479a7a Fix #520. Print influxdb's version in the log
b2f229e update the changelog
9f95a44 Fix #539. count(distinct()) with fill shouldn't panic on empty groups
ad7bfa7 rename the test
4813ddb fix the test
c3ab1bc add a shout out in the changelog
51306db Close #505. Expose the version in the response headers
1ea9695 Fix #536. New nodes should add old shards without panicing
04256a8 Revert "Fix #526. New nodes should add old shards without panicing"
ac98817 Fix #526. New nodes should add old shards without panicing
2b61fee Fix #538. Don't panic when the same series is written with different columns
bc61f56 fix the order in the log
4d6d7ff Update
730d4d4 Merge pull request #535 from influxdb/fix-535-wal-hang
a170204 wait more than the group by time
f202547 Fix #535. WAL replay hangs if the remote server crashed before sending response
a90a0b8 fix some log statements
3984a1c Fix #532. Don't log graphite connection EOF as an error
4a1f0be Fix #516. Close WAL log/index files if they aren't needed anymore
0f17a55 fix a test that breaks when multiple shards are created
e8ffd43 Rename arithemtic.go to arithmetic.go
d8348ea Update CHANGELOG for v0.6.2.
383d79a update the changelog
dec897e Fix #518. Filter Infinities and NaNs from the returned json
ff910f0 don't keep growing the slices
9d79c47 Fix #522. Committing requests while replaying caused us to skip log files
15d7228 fix #512. Group by should respect null values
0abd1c8 remove unused code
bb7769b Resposne => Response
5d5c997 Fix #369. Fix some edge cases with WAL recovery from crashes
7bc4199 add more files to .gitignore
c13e8b9 update the changelog
8db4071 Remove a Println
f98af1e Merge remote-tracking branch 'refs/remotes/origin/pr/511'
48dfffe Remove some printfs
7faf680 Fix #152. Don't automatically create databases when creating database users.
1865d29 update the changelog
f831e80 wait much longer for the continuous queries to run
dfbdf27 Add ChangeClusterAdminPassword and AuthenticateClusterAdmin
7ae7467 fix #508. Don't replay WAL for servers with no shards
fcf737d fix #507. Cluster admin passwords don't propagate to other nodes in cluster
fc21327 remove debug statements
4181918 add a tool to help inspect raft and change hostnames
b127834 use a separate db in the test
7e6ea76 Fix #503. Dropping series should get rid of all points
f2a8232 remove unnecessary if statement
d0269a3 Fix #500. Support `y` suffix in query time durations
29c55c1 Fix #501. Writes with invalid payload should be rejected
4ef7faa give a shout-out to Julien
deaa954 wait for the server to sync
3642349 update the changelog
1641d26 Fix #495. Enforce write permissions correctly
75741e9 update the changelog
d75c71a Merge remote-tracking branch 'refs/remotes/origin/pr/490'
9886824 add a failing test for #490
9be6a34 update the changelog
89df296 Merge pull request #491 from freeformz/specify_default_password
10c9838 update the changelog
d054713 Close #483. Return 409 if a database already exist
85e5072 add a failing test for pr #483
5a85dae add ChangeDatabaseUser to be able to change db user passwords, etc
23c519f Pull the initial root password from the environment if the key is set.
48590dd Looks like the wrong name / description
90ce9a8 delete the spec directory
88993da print the Ascii art to stdout only
a5e0f18 fix #431. don't log to stdout if a log file is specified in the config
8143592 Merge pull request #486 from influxdb/fix-486-remove-extra-columns
8891329 Close #477. Add a UDP json listener
7ddcf8b Remove column values that are to used in the target series name
f166903 replace casting with float literals
d621c79 make ToMap work with any ApiSeries
47a0983 update the changelog
483952d Fix #469. Dropping a db should remove all its CQ
fcfa518 add more logging
2a4d997 Move RemoveAllContinuousQueries() and AssertContinuousQueryCount() to server
01754f7 modify the way we determine whether we're in sync or not
d9835c8 fix #462. Add a test for regex with dollar sign
22eefb3 add a test for equality to negative values
ca17a54 update the changelog
1f91089 we don't need to reuse the ProtobufClient anymore
ebf4b07 Fix #478. User should be able to change raft or protobuf string in cluster
6e70b7e fix some tests in the test suite
078d23b don't write locally if the shard doesn't exist on this server
b284252 just store the raft connection string
12da880 Refactor the raft server
353a386 stop the readResponses loop on close
1fa35dd Store a pointer to local server in cluster configuration
1c09126 remove some unused ClusterConsensus functions
08d9699 add a way to change raft and protobuf ports on the fly
7b323e7 don't precreate the shards in the integration test suite
4866197 merge InfluxJoinCommand with AddPotentialServerCommand
876b059 Fix #419. Allow hostname changes in single node cluster
fe6e5df change the default value of the MaxResponseBufferSize to 100
7c17462 update the changelog
a44bc4b fix the build on 386
dbf0ac5 update the changelog
c0eab7a Merge pull request #473 from influxdb/fix-patricks-crash
c335f71 Merge pull request #472 from influxdb/fix-323-continuous-queries-should-guard-against-data-loops
f38e0a5 Fix #323. Continuous queries should guard against data loops.
a3da8b9 remove some debugging statements
5d22974 fixup! make writes synchronous for continuous queries
79b0fbe make writes synchronous for continuous queries
86c469e add a way to set the user permissions in the client
2a89d9d reduce the default response buffer size
41c7bdf some memory optimizations
b3b3ded don't write too much data at once
ffb2764 Don't do the cross product if we don't have to
3d0b1f7 fix a brittle test
93c73fb commit all series in one request
1bdea2e Use prefix trees instead of hashmaps to keep the engine state
0d35128 just write as root in the test
9362a10 We don't need to repeat the command name all over the place
f3031b4 Fix #471. User should be able to update permissions
9896a72 Update
42efc61 Update
08821ae Merge pull request #467 from influxdb/fix-466-allow-any-characters-in-column-names
0bb06d5 Merge pull request #468 from influxdb/fix-447-support-at-in-usernames
89aa0c6 Fix #447. Update username validation to only disallow %.
a064e61 Fix #466 - allow any characters in column names.
2d982f6 update the changelog
7e25c1b Merge pull request #463 from influxdb/fix-267-allow-new-characters-in-series-names
d454a8e Fix #267. Allow series names to be escaped
e671755 Merge pull request #461 from influxdb/fix-458-continuous-query-sequence-numbers
ecf5f28 Merge pull request #457 from influxdb/fix-457-delete-series-with-capital-name
4ad142f Fix #458. Correct sequence number calculation during continuous query interpolation.
182dba0 Condense test case.
276837a Test the sequence numbers more explicitly.
03435de Test for correct sequence number assignment when interpolating series names in continuous queries with a group by clause.
c330208 fix GetQueryString() for select queries with regex
462d6e9 don't try to get the last log entry if there aren't any
e1db2eb fix #457. Deleting from capitalized time series doesn't work
a7b0475 Update changelog for 0.5.9 release
312640a wait for data to replicate in the test
75024b8 rename Result to OperatorResult to avoid name conflicts with gocheck
21fdc86 update the changelog
367d4a5 add continuous queries functions
7ae531a fix #455. Comparison operators should ignore null values
04ad35b clean up after the test finishes
c500ea2 add a new db for the null columns test
da30aa7 make paul an admin :)
49b3627 update the changelog
5d02db6 fix #456. Continuous queries failed if column had null value
a9cb07d clean up the test and start using the go client
caf6bf2 Change log level on message of not sending points because there are 0 to yield to debug
9ca6330 Fix #446. Check for (de)serialization errors
de8da25 Update
1b2bca5 update the changelog
93e2571 Merge remote-tracking branch 'origin/pr/434'
13b2de3 update the changelog
28f7295 Merge pull request #451 from influxdb/calculate-summaries
7136585 fix the test
9008c76 flush the aggregator state as soon as the next bucket start
1cf9f98 simplify the logic of distinct
85dfd7a Merge pull request #450 from influxdb/fix-449-heartbeat-timeouts-cause-lockup
a37be46 Fix #449. Make heartbeats use a new buffered channel for the response for each reqeust
9ad968c Merge pull request #8 from ves/master
494584d fix typo
2b85296 add another test case to the GetQueryString test
15289d0 update the changelog
1eb0e9a Merge pull request #244 from influxdb/string-from-ast
d61eb7f remove the stupid pprof that was in the WAL test suite
915084a take a shortcut if we get an integer value
ff247d9 remove an unnecessary test
2603391 reconstruct the query string from the ast
e5ad6e3 only report success and failure on change
9111691 prepare for a release
7f68199 update the changelog
9192690 fix #439. Report the right location of the error and add pretty print
407da72 add travis notification to hipchat
876de6b fix #437. Note the previous fix in 6cb9413 broke other functionality
afb76a8 fix #432. Create a checkpoint before executing a delete query
6cb9413 fix #437. Queries with negative constants don't parse properly
8d810e8 remove a whitespace
5295f19 Revert "Fix #433. Remove custom interfaces endpoint."
7b969c1 Revert "Remove unused import. /me shakes fist @toddpersen"
e829508 update the changelog
826e9f9 fix #442. Fix some bugs in shouldQuerySequentially and add a test
545cc05 use point batch size from the config file
8d7e91a Change logging of query to be an info level event
9a2f9f1 Merge pull request #429 from influxdb/fix-tests
d9fdc28 More robust test suite
583b08d simplify
378723a Be less verbose when servers go up and down
fc8b35d get rid of tip, it's failing on travis
bde7e91 log an error if the graphite plugin can't write data
ce00440 make the heartbeat backoff configurable
b88f760 use the same query spec
e3c55bd start the protobuf server after raft has initialized
69c1901 add a way to tell if the cluster is in sync or not
d199866 simplify the recovery logic
d3eba5f Update changelog
0d29548 Merge pull request #441 from influxdb/fix-440-heartbeat-timeouts
2aa5933 Fix #440. Update protobuf client to clear out heartbeat requests after they've completed
75a81e0 fix another bug with initial recovery
673e66d use shard ids not server ids
7978283 Add log output on error with openeing shard
cba2032 Remove unused import. /me shakes fist @toddpersen
3408c32 Merge pull request #427 from lucapette/fix_typo
f6fc73d Fix #433. Remove custom interfaces endpoint.
46a874d Merge pull request #435 from mitbra/make-clean-typo
c25e816 typo in 'clean' target
6a7839c fix #423. rename config sample config.{toml.sample => sample.toml}
56afa38 Add output on heartbeat error
4502961 Add the request to the running request object in protobuf client so we can output it if it times out for more debug info.
ffaf089 Change log output for buffer size on query to debug. Too noisy on Info.
8ee3131 Fix typo
9086c99 Fix config parsing so we actually get the per server write buffer size
5371dc9 fix #328. Join queries with math expressions don't work
8756940 bump the version in the changelog
c163db8 update the changelog
08ab09a Merge pull request #392 from influxdb/fix-392-different-columns-across-shards
2785782 fix #392. Different columns in different shards returns invalid results
0748eda wip: add tests to illustrate #392
998bf82 Merge pull request #424 from influxdb/fix-421-api-read-timeout
caf3b45 Fix #421. Add read timeout to API to make sure we don't keep thousands of connections open needlessly.
dbbea8d fix #415. Include the database when sending an auth error
3973b5f update the changelog
a04aed4 Merge pull request #413 from influxdb/fix-413-small-group-by-time
683a234 fix #413. Don't assume the group by interval is > second
03decd4 no need to copy to a buffer then reset it
b303a1d fix #416. Improve the time it takes to drop an entire database
9dce95e update the changelog
4d18a66 Merge pull request #310 from influxdb/fix-310-request-multiple-series
4acdc7b fix #310. Request should support multiple timeseries
c116f13 fix some tests that were broken from earlier changes
fa9361f add some packages to the makefile
f26ba5f update the changelog
800ecf0 oops move StartProcessingContinuousQueries few lines up
8ca9836 fix #407. Start processing continuous queries after initializing the WAL
5a230b9 Revert "fix #405. Start processing continuous queries after initializing the WAL"
c49f110 fix #405. Start processing continuous queries after initializing the WAL
886a036 fix #390. multiple response.WriteHeader when querying as admin
bf9975b fix #408. Make InfluxDB recover from internal bugs and panics
cd7c639 fix #401. No limit on the number of columns in group by clause
bda7392 change the order of the changelog
06f4cdd fix #405. Percentile shouldn't crash for small number of values
7fc300d update the changelog
16cbdba fix #404. This is just a test the real fix is in ddd7e0e
132287a fix #403. Filtering should work with join queries
44ff5f4 implement ApiSeries interface
c46ac82 update the changelog
3323cb2 add an option to repair leveldb databases
0866722 close the leveldb if we get error creating LevelDbShard
9b5027a use interface so we can reuse the methods with *influxdb.Series
ddd7e0e don't panic on filtering errors, instead log an error
ed16351 DRY
b664bc0 print the version of leveldb when -v is passed to influxd
d724617 remove the benchmark tests which aren't used anymore
8ccc581 refactor the benchmark test suite to use influxdb-go
b315c89 add another config option to generate the right schema
d5af821 support an optional time precision argument in Query()
aee5200 add Ping() and AuthenticateDatabaseUser()
740f0e0 fix #7. Make it possible to pass an http client
845afb5 fix #398. Support now() and NOW() in the query lang
1967629 more fixes for issue #341
d39646a update the changelog since leveldb upgrade didn't make it in 0.5.4
93e93fa update the changelog
e3db954 fix #362. Limit should be enforced after the aggregation
7d08c0b more fixes for issue #341
b127201 fix #394. support count(distinct) and count(DISTINCT)
0f360a3 add the leveldb upgrade to the changelog
f1df3dd update the changelog
ee0277a Merge pull request #341 from influxdb/fix-341-query-memory-consumption
0e8a5ce refactoring to make the code clear and simple and fix some bugs
c3bf487 fix #341. Limit the amount of memory taken by query
f959c43 update the changelog
bb2f96e fix #389. Filtering shouldn't stop prematurely
3a94684 upgrade to leveldb 1.15
ca28d1d fix #386. Drop series should work with names containing dots
767ba83 update the changelog
8c5bb2a Merge pull request #379 from influxdb/fix-379-boolean-columns-where-conditions
d369e51 fix #379. Boolean columns testing in where conditions
f76762d Merge pull request #370 from influxdb/fix-370-shards-limit
67545b6 introduce some sleeps to make sure the data is updated
aa61caa fix #370. Filtering and limit should be enforced on the shards
e4335cf fix the shard boundaries test
3f543ea remove unused code
1d4d323 remove an unused func
0769ff9 Add tests to verify that cluster admins can issue deletes.
e379a2f Fix #381. Cluster admins should be able to issue deletes.
72fba59 fix #378. Index should return immediately if there are no new requests
491924d add google-pprof cpu profiling
d1e15fa update the changelog
92a7d37 oops, fix a bug introduced in 1772fe9. Tests FTW
36582ec update the changelog
1772fe9 fix #360. Store and recover continuous queries on startup
c8f88c5 use the ContinuousQuery structure instead of an interface
21f089d increase the timeout
c5e2e2c don't close the server if it hasn't started yet
47bbb67 change the default log level to info
5c38e52 update the changelog
e2490a2 fix #371. Seris index isn't deleted when the series is dropped
d37d89d update the changelog
1567c69 fix #331. Allow negative time values
99f6dc7 update the changelog
6b890e9 fix #355. Return an error on invalid time strings
40e7d08 panic on error
33414d7 print the number of columns as well
fb2ece5 panic if the WAL cannot append the request to disk
fdd0118 fix the shutdown logic when we're not profiling
b8dbac8 renaming
49ee059 fix #367. Influxdb won't start if the api port is commented out
bf883e8 update the changelog
7654856 Merge pull request #342 from influxdb/fix-342-delete-bug
87f4964 test dropping database as well
5686684 fix #342. Make sure we create a checkpoint before deleting data
9246361 Add an api to create a wal checkpoint (i.e. bookmark and index)
089cb6f add a tool to fix the index of a raft log file
10e1e0e formatting
7bcdc9c Merge pull request #6 from kuba--/master
1ff6b65 Fix typo
9ee3140 ignore more files
a15a2a9 update the changelog
af7455d checkin the source code of goraft
648206a oops, fix a typo
2366a9e make different submdoule version doesn't cause the release to stop
71660f8 change the goraft commit that we use
952ce0b Merge pull request #336 from influxdb/fix-336-raft-config
549eff3 Merge pull request #359 from influxdb/fix-359-log-compaction
7696d6e add goraft as a git submodule
9c22aab add a test to demonstrate a bug in the log compaction
9b1f922 ignore more files
1fd64f9 make the raft election timeout configurable
8210516 fix the version of the last 0.5.0 release candidate
0d487ed update the changelog
a11a11e Close #293. Merge branch 'pr-293'
5378787 increase the sleep to make sure the shards are created
1043101 modify the configuration to match this example
af111ac remove a printf
7e957ec implement graphite listener
ced0078 add a graphite listener test
0fb8014 add more logging
eb292f7 change printf to debug
c056cf5 Merge pull request #357 from influxdb/fix-340-out-of-order-commits
447969c Fix replay so writing points while replaying doesn't cause a problem. Now the replay loop will work from the last known written request.
50edea2 Add log output for write buffer replay.
b01b6bd Increase raft election timeout.
1c38157 Fix list series query that I broke a few commits ago. Effffff.....
05c86f1 Merge pull request #352 from influxdb/sort_points
38c1bda Use SortPoints method that was already in protocol extensions
b321edb sleep while waiting for the other server to recover
830c72a sort points before writing them to reduce the number of logged requests and replication requests
5f37652 Make list series requests use a buffered channel to not drop responses.
86995e6 Make write on cluster server have a buffered channel so we can be sure to read the response.
c489ee2 Create new request objects when sending writes to servers. This will ensure to avoid collisions on the request id in the protobuf client.
78c82f4 Make heartbeats use a buffered channel.
5acb994 Merge pull request #335 from eckardt/fix_multiple_vars_in_continuous_query_interpolation
a77692e Merge pull request #344 from lucapette/patch-1
056c8c3 Fix typo in conf file
160bd6c Script to randomly write batches of points over a time interval.
d3124b3 Merge branch 'pr-318'
7bda031 change the default log level to debug
df84b7f fix few bugs introduced in dbe76df
1fdf515 Fixes regexp for interpolating more than one column value in continuous queries
f570682 DRY the parser a little bit, we already have a definition for SELECT
f2e9c8a fix some log statements
e2adcf1 Close #318. Support EXPLAIN queries
4da21dd add some explain queries tests
dbe76df partial fix for issue #294 processing some queries in batches
e798c98 fix a broken test in the parser package
9fdf025 don't forget to include the version.go in the source tar file
36932ad remove the version from the influxd.go file
38b37bf update the changelog
4f30e59 Merge pull request #333 from influxdb/fix-333-crypto-bug
0bc99d1 fix #333. Readable error msgs and don't create user if pass is invalid
fe2503f sleep for the data to replicate
87a284e remove redundant test
b4bdb45 move the user manager interface to the api package
f9f00c6 set the process to nil after we kill the server
603621e add some comments
b5f68b7 update the changelog
b4babbd Merge pull request #302 from influxdb/fix-302-negative-time
6b178a5 Merge remote-tracking branch 'origin/fix-315-commits-not-in-order' into fix-302-negative-time
471039a fix #321. Fix a off-by-one bug in the splitting algo
7e659b9 make sure that a shard endtime is exclusive when we search for shards range
3a5adb7 fix time stamp rounding when we create a new shard
94f06b7 add some logging
89e7eae fix the sleep since the process takes about 5 seconds to startup
3c664f4 fix #302. Should be able to set negative times
a33bb37 fixup! we shouldn't be returning error if the function takes a response channel
64571fc Clean up test spacing.
c54588d A massive group by test with real data.
fe10d5b Output the shard's start and end timestamps on String().
ef92908 Add StartMicro and EndMicro methods to the shard.
e698a56 sleep for a few seconds to give a chance for raft to replay the log
e1b996e more logging
5db312d fix a compilation error
5c56bfa change the name of the series in the test
2fe6fdc fix a bug, use value comparison instead of pointer comparison
ad99747 we shouldn't be returning error if the function takes a response channel
c33f94c remove some logging statements
359584d Drop the max delete size from 1M to 64K.
6fab943 Revert "panic if we ever receive commits out of order"
bdb6cf4 fix a compilation error
4ede6f8 fix #233. Put back the code that limited the delete batch size
e0e7144 cleanup default start and end time code
710980c fix a typo
b3c35ff Fix #315. Make delete queries not use WAL. They should instead use Raft to replicate.
6dc127b add a test to make sure invalid time group by interval don't crash the server
39cec27 update the changelog
e5b1486 Merge pull request #324 from influxdb/fix-316-access-denied-goroutine-starvation
2a780d3 Fix #316. Make shard hold access denied responses until all response channels are emptied out.
7ba3347 remove some unused constants
d70972c update the changelog
7b74e5d Propagate the error back and assert on the body of the response
be7137a Merge branch 'plumb-nqe-fail' of
74218be uncomment some of the engine tests that tested invalid queries
db5ba71 update the changelog
dd55e2c Merge pull request #301 from influxdb/ref-count-on-open-shards
afd2d8c Updated ref counting to remove accessed shards from the list of ones to close. Updated the close to close as many as it takes to get below the leve
e39508f Add test for max shards and fix bug in closing out old one
2bce283 Fix bug where I was deleting the shard reference before pulling out the shard to close it
0d6e2d6 Add ref counting to which leveldb shards are open so they can be closed after references released
386d3d6 Allow NewQueryEngine to return an error
0e98711 accept any request number initially
20a921e panic if we ever receive commits out of order
378d185 fix a bug introduced by my refactoring
a2f0bf8 flush on every write in the tests
41b776c don't do this twice, 5 seconds is good enough
73648b5 move the sleep outside the loop in the test
d029273 more logging
10b2820 fix another bug in the wal, not all requests have series
564c0f1 fix the broken tests, now recovery happens after server id is set
0c7542d fix #312. WAL should wait for server id to be set before recovering
6a3b16e more changes to the changelog
ab430f8 update changelog
8f95ce4 update changelog
c5633c6 remove some whitespaces
417f035 wal should go in the opt directory with the rest of the data
348d5b5 ignore more files
43b596c some refactoring
4b974ba fix #309. Don't relog the delete queries
2cb3124 rename import
f759d37 add a test to make sure we don't relog requests
90566ac graceful shutdown
134b4c4 more logging
ea5f65e update the changelog
417c23f fix more bugs in the wal
66c0f91 refactor the wip and fix some bugs
d22874a Update
402d7bc Merge pull request #308 from influxdb/fix-305-shard-ids-not-unique-after-restart
1da8fef fix bug where writes from the protobuf connection were incorrectly getting logged to the wal and buffered.
a3c83b6 Fix #305. Ensure that the shard id is set based on the number of current shards + 1. 305 was only occuring if a restart after a Raft log compaction happened
58cdd6f Merge pull request #298 from influxdb/fix-298-limit-with-multiple-shards
f3f36e0 fix #298. fix limit when querying multiple shards
91dfff2 prepare the changelog for a release
10596f1 Merge pull request #292 from influxdb/fix-186-max-open-shards
a09abd4 fix #286. Make numbers of open shards configurable
10fc4f9 fix the configuration parser
6896f17 update the change log
1c9e14f fix #287. make the lru cache size configurable
1565e55 make sure we close the iterators after we query
6653b40 Update changelog for 0.5.0-rc.3
2409047 Merge pull request #290 from influxdb/fix-69-column-aliasing
0d52464 fix #69. Support column aliasing
5bb5161 add support for function calls aliasing in the parser
a3c3c79 Merge commit 'df2056de8a9644f13b84540fadf123e55ef238d9'
df2056d fix #285. fixing some problems with req num roll over
0fd2a4c fix the configuration test
d626216 Add required files.
0d5d05e Add more tests.
b883b1f update the changelog
706d58c fix #288. Sequence numbers should be unique per server
6e3fb51 fix the time precision in a test
23412f1 quote the test regex
20d334c Add in max open files per shard
8f93eb9 fix #284. WAL should recover from files ending prematurely
9513a61 Add setup script for the benchmark tool
47058eb Increase default buffer sizes to make replays less likely.
e4c039b update the changelog
06070d2 Remove old cruft of leveldb_datastore. Add close method to the new shard store and make server.Stop call it. Remove old cruft on protocol, make request number a lower id.
fc12c75 fix #281. make build should replace the version string
2e41160 Fix bug where the last write gets dropped and no other write follows on the buffer. Was causing relpay to error out
e9d39ca Ignore benchmark log
64383b5 Fix write buffer replays to properly play requests in order and not drop any mid-replay.
115d5d1 Fix #283. Dropping shard and restart in cluster causes panic.
e314813 set the request number on the request when we're replaying
8e53366 update the changelog
21618be break out of the proper loop
245174a fix #279. Limit not working for regex
4698a4fb speed up the test initialization
4cf4ad3 be consistent in the chanagelog format
bc6e5ea update the changelog
622fcbb fix a typo in the replay logic, we should read from the new file
c42e613 fix #274. open a new fd for replay
b931784 add some logging to log.go
c1019cd log a message when we rotate the log
e0fc50d make closing a wal safe
165a6a0 add some logging to the wal recovery
e4a3c53 add a test to illustrate corrupted wal
12f6a0d sleep before exit to print the log messages
f74c9d3 Stub out the burn-in suite.
6f5eae1 Fix #277. Ensure that duplicate shards won't be created for a given time.
f951814 fix the go build tag
10dbcdc Fix engine test to not fail after half past the hour
3ec1502 fix the renaming of the latest rpm package
f4da084 oops, change the rpm version only
d9c7a26 fix the package version in the makefile
6cd0bc9 Remove a bunch of debugging print lines
0c41aeb Fix #265. Add support for series name interpolation in continuous queries.
86e21f7 Merge pull request #269 from influxdb/create-cluster-interfaces-and-refactor
9eab588 sleep after writing data in the integration suite
eb8d28c Merge branch 'master' into create-cluster-interfaces-and-refactor
d0839e2 fixes #258, fixes #250, fixes #216, fixes #166, fixes #165, fixes #164, fixes #132, fixes #103, fixes #65
9ff7ce8 0 write timeout should be treated as no timeout
a2865bd fix the api test suite
50491fc fix the coordinator test suite
a8f2330 fix the coniguration test suite
86fd769 remove a test that's not relevant anymore
dbac16b sleep little longer
00b85b1 expect no series instead of empty series
893f4d7 Comment out old datastore tests
90335f7 Make protobuf client reconnect on error.
da2b3e5 remove the if statement to actually yield empty series
8e1261b series with no points don't return anything
3679e41 sleep longer to make sure data is deleted
c3fb89d Fix test that does a big group by query against multiple shards so that it doesn't break every other day.
036ea3c Made protobuf client always send end stream responses even if dropping other data on the floor.
a76532d Fix race condition where tests would freeze up randomly.
099694e Make recover from WAL look for raft server names to set the local id.
ae9c545 Add additional error forwarding to request handler that will close the response channel out.
417582d Removed retry logic and extra cruft from protobuf client for something simpler.
679403b Fixed bug where raft snapshots were storing the local server id, which isn't shared state. Updated initialization and recovery to set the local server id based on if a given server's Raft name matches the raft name set locally.
bcb2f25 Update engine to not yield empty series. Make sure that errors are sent in the response.
aef2f1a Add small delay after POST and DELETE in tests to make sure write buffers have cleared out.
18abe42 Fix issue that was causing cross shard queries to hang. Queries that don't aggregate locally weren't closing out the seriesWriter.
58db5c6 Make sure we drop the continuous query at the end of the test.
b9a337a Remove test output.
6fa16b4 Fix bug where continuous queries were causing the inflight data to get changed. Result of buffering writes now.
df74078 Make sure the continuous query test deletes the old queries even if it fails. Paul says "use defer, Todd, use defer"
11ee648 Make list series engine not yield empty response, which was breaking the test when shards didn't have any series.
2b6dd37 Lock the connection when setting it on the protobuf client.
e8d2352 Handle access denied responses to close out stream. Close the seriesWriter when a list series query is done.
1853812 Make an error on request to another server send the message down the response channel (non-blocking) so that go routines monitoring the channel and not the method return can stop.
faa33b0 Ensure recovery from WAL connects servers.
3917cc5 Remove old data replication test since it's no longer valid.
e9560b0 Fix for server failure scenario test. Make reconnects non-blocking. Update heartbeating to timeout when waiting for a response.
4e2c9b8 Comment out the test failure with deletes. For now deletes are not highly available and require every server in the cluster to be running.
6232c4f Increase timeouts and heartbeat intervals on test and sample config files.
c0c84b4 Make heartbeat interval and backoff public so they get serialized. Fixes 100% CPU issue that was being caused by heartbeats getting sent out without sleep periods.
0892d34 Make protobuf client and request handler write protobuf length and data in one call to write to avoid race conditions
1154adc fix the tests to expect no series instead of series with no points
dd3f024 fix a bug in single point queries and better error handling
b2c66d3 update the release script to pull and clean the repo before building
b5321d4 remove println
cdfff9d Add code to precreate shards in the integration test
633459f fix compilation errors in the tests
9aa1194 add a HasAggregates function to SelectQuery
3ba4673 fix a test
d65bdf4 move the log channel to WAL and fix the wal tests
57f2cf2 close the write log on exit
2ebbb69 fix few bugs with the heartbeat logic
fe12f17 Fix destructive queries. Add logic to have non-aggregate queries use a local engine.
51f03d9 Fix #250. Continuous queries now get auto-incrementing sequence numbers assigned per timestamp.
3701403 Wired up recovery from log on startup. Fixed bugs with commits on request numbers
5d3ef2a Add check to ensure there are healthy servers to query and return error if not.
4d0fd93 WIP: Add write buffer to wrap writes to servers or the local datastore and replay from logs if it gets backed up. Errors on test because Shahid didn't update the test config files or set defaults on timeouts? /me shakes fist
f54b29a handle heartbeat requests
37daaa2 don't block forever when writing to remote servers
606fe19 query healthy servers only
29f21a1 make heartbeat interval and timeout configurable
c51118d implement recover from last committed request
104fdf3 make the log file rotation thread safe
1bd8dc8 Change wal.Commit to take serverId
36b5385 Fix #257. Make sure we're enforcing the validity of continuous queries on creation.
74b1519 Add method to cluster.WAL interface to recover a server from last commit.
2b9181f Fix #217. Make sure permissions error messages aren't getting swallowed.
91748e0 delete some printfs
6f19be1 fix some nasty bug in the batching logic. /me shakes fist at paul :)
f02bd8c fix the series equality checker
e297ca9 support aliases when yielding points
de3158d limit replication factor to be a max of the number of servers in the cluster
3af5ee2 Add logic to break up responses into smaller chunks if they're too big
744445d Make server create shards for future time frames before they hit
6bbbe0f Update default RF to 1 and short term shard size to 7d
5a55fd0 Fix bug on group by using day
5a5e8e5 Change the constant for the first lower case char to use a char instead of a number to make Johnny S. happy.
356a226 Remove old ring location helper.
00c1f9b Make config use the common parse duration to support days
c3451f7 move ParseTimeDuration to common and make it public
931154d remove a print statement
ef48353 fix more bugs in batching
50c590f fix a bug in how the points are split into multiple shards
e720e66 Wire up queries to regexes. Fix bug in limits on regex queries
5111773 Add test that does a single write that needs to be split to multiple shards.
e2274cd Wire up save and recovery of cluster config for Raft.
5d68db9 API endpoint for continuous queries should return simplified results.
2a13ffa Remove old stuff from cluster config
3924ef8 fix the join and merge bug
b10dd38 Wire up API for getting cluster servers, and creating and dropping shards.
983baf4 refactor the code
f22924e fix some tests
35945eb add some comments
11708ca fix the time precision in the engine test
8eb8ec9 fix the wal interface
2992444 don't set the sequence number in the test
55a9f78 make the id field in the request optional
e6eb494 update the config files
f7142f7 more cleanup
5d3586d more refactoring and fix the case when the yield function errors out
52266f2 some refactoring
007b0c9 implement log compaction
0425400 make the number of requests per log file configurable
6ec5995 some bug fixing
4e62287 force index creation on close
68e73e9 add log index
7fdbb75 make the index entry size configurable
0a5fd2c create an index for the log file
7605782 fsync when we create the bookmark
25aa676 flush every configurable number of requests
c453e55 implement configurable auto bookmarking
bda97a2 bookmark and recovery
c18d664 fix a bug in the skipping code
daea800 make the flush and bookmark configurable
498e091 assign sequence numbers to points.
3ad2a5e be consistent with the rest of the codebase
b052f01 wip: implement write ahead log
193dd9a wip: implement write ahead log
ae6fdc8 add Decode method to protocol.Request
7bad412 wip: whatevs
f39a47c slight changes to the wal api
ffeed32 fix #103. Add location of wal to the configuration
95d7831 Add /cluster/servers endpoint to get server information.
8947bb7 Fix bug where group by times that are less than duration of a shard but would require a split would not be calculated properly.
31bfc03 finally got some engine tests to pass
258336d Wire up queries that require aggregation across shards.
3afc9f7 Wire up ascending queries that span multiple shards.
b41e77c Made RunQuery on coordinator take a SeriesWriter interface for better testability. Hope you're happy John.
79efd8e Wire up drop database. Remove cruft from coordinator, request handler, and the protocol.
570a6100 fix the api test
2b2d587 fix the engine test suite, it runs now
17b7912 some refactoring to make the engine suite compile
3bce8f4 wip: move the engine tests to the integration test suite
a8c3f85 fix the coordinator test assertion
15209ad Remove old handling code for replication write and proxy write.
9f51e96 Wire up drop series query.
ca1d9bf fix #189. Use name instead of username when listing users
a3ebd57 Add yield func type to clean up method signatures
f66d168 Wire up delete from series.
d46e29a Remove old ListSeries functions
9edfbf9 Wire up the list series query and remove the passthrough processor.
8befa09 Wire up queries against remote shards.
d547518 Remove call to SyncLogs in server since that's going away.
3465de8 Wire up logic to get shards for a query.
bf13938 Wire up replicated writes, creation of shards across cluster.
a76c63e Wire up on the fly creation of shards, get of shards for writes and writes in a shard.
177c976 move ParseTimeDuration to common and make it public
6ef724c move the parse duration code in its own func
1c48607 kill a file that we're not using anymore
88231d8 Move common code in the WriteSeries functions to writeSeriesCommon
b86fca4 Merge branch 'WriteSeriesWithTimePrecision' of
2acc30f Added a method to write series with a specific time_precision
2eb922c User common.User interface instead of coordinator.dbUser
a975d94 Merge pull request #251 from influxdb/fix-246-add-details-to-user-index
3c0c605 Fix #246. Add user details to index endpoint.
7e24059 Update changelog
8ece0f6 Merge pull request #245 from influxdb/fix-243-get-database-user
e9d3623 Change `username` to `name`.
5d30d2f Add API call to return user details.
6d44c5d Clean up variable naming.
c921776 Added limits for non-aggregate queries.
d59238c Fixed aggregate queries.
cc52247 Wrote and wired up where clause. Refactored engine to be used by a shard instead of by the coordinator. Moved filtering over to engine.
eba8f5a Wire up local shard and simple querying without limits, aggregates, or where clauses.
111403a WIP: wire up some of the query functionality in coordinator and create query spec object in parser.
6c2a673 Wire up writes in leveldb_shard and the cluster local shard.
9135ad8 Make request shouldn't call connect. This happens when the server is added to the cluster config.
27faa01 Make cluster configuration set the connection on the server object when adding it.
288f468 Fix server startup to wait for local server id after raft server is started
16b0052 Merge branch 'master' into create-cluster-interfaces-and-refactor
8a075bc Remove cluster servers from Shard since it's not used
00ff999 WIP: more code around writes from coordinator into shards. Shards to local store or server.
13aafa1 Add interface for WAL and stub out.
eb30e9a WIP: refactor so coordinator.RunQuery is the single entry point for running queries.
a51d41d Make block size 64k
7c4846d Add initial parts of the sharded leveldb store. Stub out shard management in raft server. Update server to work with new cluster configuration.
f6583bc Update Shard Query interface. Add getting shard by series and time
625a419 Remove extraneous fmt.printf and add some loggigng
17ae70b Fix coordinator test so raft server and cluster config use same configuration object.
3df7afa Add cluster interfaces, refactor cluster config and server into cluster package.
74febc7 add a test
3a0cc34 add a go influxdb client
89bafef add .gitignore
c91885e Initial commit

git-subtree-dir: src/
git-subtree-split: 63cc69f1dfb12f35ad8a3e4fb4b1115d3a1edc91
@jvshahid jvshahid was unassigned by Dieterbe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 4, 2014
  1. @Dieterbe
Commits on Mar 6, 2014
  1. @Dieterbe

    graphite ingestion fixes

    Dieterbe committed
  2. @Dieterbe

    comment fixes

    Dieterbe committed
  3. @Dieterbe

    small graphite fixes

    Dieterbe committed
  4. @Dieterbe
Commits on Mar 7, 2014
  1. @Dieterbe

    better comment

    Dieterbe committed
Showing with 210 additions and 0 deletions.
  1. +6 −0 config.toml.sample
  2. +171 −0 src/api/graphite/api.go
  3. +21 −0 src/configuration/configuration.go
  4. +12 −0 src/server/server.go
6 config.toml.sample
@@ -23,6 +23,12 @@ port = 8086 # binding is disabled if the port isn't set
# ssl-port = 8084 # Ssl support is enabled if you set a port and cert
# ssl-cert = /path/to/cert.pem
+# optionally enable a graphite (carbon) compatible ingestion
+enabled = false
+port = 2003
+database = "" # store graphite data in this database
# Raft configuration
# The raft port should be open between all servers in a cluster.
171 src/api/graphite/api.go
@@ -0,0 +1,171 @@
+// package Graphite provides a tcp listener that you can use to ingest metrics into influxdb
+// via the graphite protocol.
+// it behaves as a carbon daemon, except:
+// no rounding of timestamps to the nearest interval. Upon ingestion of multiple datapoints
+// for a given key within the same interval (possibly but not necessarily the same timestamp),
+// graphite would use one (the latest received) value
+// with a rounded timestamp representing that interval.
+// We store values for every timestamp we receive (only the latest value for a given metric-timestamp pair)
+// so it's up to the user to feed the data in proper intervals
+// (and use round intervals if you plan to rely on that)
+package graphite
+import (
+ "bufio"
+ "cluster"
+ log ""
+ . "common"
+ "coordinator"
+ "io"
+ "net"
+ "protocol"
+ "strconv"
+ "strings"
+ "time"
+type Server struct {
+ listen_addr string
+ database string
+ coordinator coordinator.Coordinator
+ clusterConfig *cluster.ClusterConfiguration
+ conn net.Listener
+ user *cluster.ClusterAdmin
+ shutdown chan bool
+type GraphiteListener interface {
+ Close()
+ getAuth()
+ ListenAndServe()
+ writePoints(protocol.Series) error
+// TODO: check that database exists and create it if not
+func NewServer(listen_addr, database string, coord coordinator.Coordinator, clusterConfig *cluster.ClusterConfiguration) *Server {
+ self := &Server{}
+ self.listen_addr = listen_addr
+ self.database = database
+ self.coordinator = coord
+ self.shutdown = make(chan bool, 1)
+ self.clusterConfig = clusterConfig
+ return self
+// getAuth assures that the user property is a user with access to the graphite database
+// only call this function after everything (i.e. Raft) is initialized, so that there's at least 1 admin user
+func (self *Server) getAuth() {
+ // just use any (the first) of the list of admins.
+ names := self.clusterConfig.GetClusterAdmins()
+ self.user = self.clusterConfig.GetClusterAdmin(names[0])
+func (self *Server) ListenAndServe() {
+ self.getAuth()
+ var err error
+ if self.listen_addr != "" {
+ self.conn, err = net.Listen("tcp", self.listen_addr)
+ if err != nil {
+ log.Error("GraphiteServer: Listen: ", err)
+ return
+ }
+ }
+ self.Serve(self.conn)
+func (self *Server) Serve(listener net.Listener) {
+ // not really sure of the use of this shutdown channel,
zorkian added a note

I think the shutdown here doesn't do much; or at least, not what you hope.

It works in the HTTP API because the underlying net/http module's Serve() method has some intelligent handling of Accept() failures:

In short it determines if it's a temporary or permanent failure, and it aborts the accept loop on permanent failures. That allows the Serve() method to exit, which ends up triggering the defer to send the true on the shutdown channel.

I haven't fully thought through the implications of shutdown in InfluxDB (just started looking at the code today) so I don't know what the right solution in API modules like this is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ // as all handling is done through goroutines. maybe we should use a waitgroup
+ defer func() { self.shutdown <- true }()
+ for {
+ conn_in, err := listener.Accept()
+ if err != nil {
+ log.Error("GraphiteServer: Accept: ", err)
+ continue
+ }
+ go self.handleClient(conn_in)
+ }
+func (self *Server) Close() {
+ if self.conn != nil {
+ log.Info("GraphiteServer: Closing graphite server")
+ self.conn.Close()
+ log.Info("GraphiteServer: Waiting for all graphite requests to finish before killing the process")
+ select {
+ case <-time.After(time.Second * 5):
+ log.Error("GraphiteServer: There seems to be a hanging graphite request. Closing anyway")
+ case <-self.shutdown:
+ }
+ }
+func (self *Server) writePoints(series *protocol.Series) error {
+ err := self.coordinator.WriteSeriesData(self.user, self.database, series)
+ if err != nil {
+ switch err.(type) {
+ case AuthorizationError:
+ // user information got stale, get a fresh one (this should happen rarely)
+ self.getAuth()
+ err = self.coordinator.WriteSeriesData(self.user, self.database, series)
+ if err != nil {
+ log.Warn("GraphiteServer: failed to write series after getting new auth: %s\n", err.Error())
+ }
+ default:
+ log.Warn("GraphiteServer: failed write series: %s\n", err.Error())
+ }
+ }
+ return err
+func (self *Server) handleClient(conn_in net.Conn) {
+ defer conn_in.Close()
+ reader := bufio.NewReader(conn_in)
+ for {
+ buf, err := reader.ReadBytes('\n')
+ if err != nil {
+ str := strings.TrimSpace(string(buf))
+ if err != io.EOF {
+ log.Warn("GraphiteServer: connection closed uncleanly/broken: %s\n", err.Error())
+ }
+ if len(str) > 0 {
+ log.Warn("GraphiteServer: incomplete read, line read: '%s'. neglecting line because connection closed because of %s\n", str, err.Error())
+ }
+ return
+ }
+ str := strings.TrimSpace(string(buf))
+ elements := strings.Split(str, " ")
+ if len(elements) != 3 {
+ continue // invalid line
+ }
+ val, err := strconv.ParseFloat(elements[1], 64)
+ if err != nil {
+ continue // invalid line
+ }
+ timestamp, err := strconv.ParseUint(elements[2], 10, 32)
+ if err != nil {
+ continue // invalid line
+ }
+ values := []*protocol.FieldValue{}
+ if i := int64(val); float64(i) == val {
+ values = append(values, &protocol.FieldValue{Int64Value: &i})
+ } else {
+ values = append(values, &protocol.FieldValue{DoubleValue: &val})
+ }
+ ts := int64(timestamp * 1000000)
+ sn := uint64(1) // use same SN makes sure that we'll only keep the latest value for a given metric_id-timestamp pair
+ point := &protocol.Point{
+ Timestamp: &ts,
+ Values: values,
+ SequenceNumber: &sn,
+ }
+ series := &protocol.Series{
+ Name: &elements[0],
+ Fields: []string{"value"},
+ Points: []*protocol.Point{point},
+ }
+ // little inefficient for now, later we might want to add multiple series in 1 writePoints request
@pauldix InfluxData member
pauldix added a note

Agreed. #310 will add that functionality to requests, which will make it possible to update the interface on the coordinator. Will definitely make things more efficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ self.writePoints(series)
+ }
21 src/configuration/configuration.go
@@ -33,6 +33,12 @@ type ApiConfig struct {
Port int
+type GraphiteConfig struct {
+ Enabled bool
+ Port int
+ Database string
type RaftConfig struct {
Port int
Dir string
@@ -125,6 +131,7 @@ type WalConfig struct {
type TomlConfiguration struct {
Admin AdminConfig
Api ApiConfig
+ Graphite GraphiteConfig
Raft RaftConfig
Storage StorageConfig
Cluster ClusterConfig
@@ -142,6 +149,9 @@ type Configuration struct {
ApiHttpSslPort int
ApiHttpCertPath string
ApiHttpPort int
+ GraphiteEnabled bool
+ GraphitePort int
+ GraphiteDatabase string
RaftServerPort int
SeedServers []string
DataDir string
@@ -214,6 +224,9 @@ func parseTomlConfiguration(filename string) (*Configuration, error) {
ApiHttpPort: tomlConfiguration.Api.Port,
ApiHttpCertPath: tomlConfiguration.Api.SslCertPath,
ApiHttpSslPort: tomlConfiguration.Api.SslPort,
+ GraphiteEnabled: tomlConfiguration.Graphite.Enabled,
+ GraphitePort: tomlConfiguration.Graphite.Port,
+ GraphiteDatabase: tomlConfiguration.Graphite.Database,
RaftServerPort: tomlConfiguration.Raft.Port,
RaftDir: tomlConfiguration.Raft.Dir,
ProtobufPort: tomlConfiguration.Cluster.ProtobufPort,
@@ -292,6 +305,14 @@ func (self *Configuration) ApiHttpSslPortString() string {
return fmt.Sprintf("%s:%d", self.BindAddress, self.ApiHttpSslPort)
+func (self *Configuration) GraphitePortString() string {
+ if self.GraphitePort <= 0 {
+ return ""
+ }
+ return fmt.Sprintf("%s:%d", self.BindAddress, self.GraphitePort)
func (self *Configuration) ProtobufPortString() string {
return fmt.Sprintf("%s:%d", self.BindAddress, self.ProtobufPort)
12 src/server/server.go
@@ -2,6 +2,7 @@ package server
import (
+ "api/graphite"
log ""
@@ -16,6 +17,7 @@ type Server struct {
ProtobufServer *coordinator.ProtobufServer
ClusterConfig *cluster.ClusterConfiguration
HttpApi *http.HttpServer
+ GraphiteApi *graphite.Server
AdminServer *admin.HttpServer
Coordinator coordinator.Coordinator
Config *configuration.Configuration
@@ -53,6 +55,7 @@ func NewServer(config *configuration.Configuration) (*Server, error) {
httpApi := http.NewHttpServer(config.ApiHttpPortString(), config.AdminAssetsDir, coord, coord, clusterConfig, raftServer)
httpApi.EnableSsl(config.ApiHttpSslPortString(), config.ApiHttpCertPath)
+ graphiteApi := graphite.NewServer(config.GraphitePortString(), config.GraphiteDatabase, coord, clusterConfig)
adminServer := admin.NewHttpServer(config.AdminAssetsDir, config.AdminHttpPortString())
return &Server{
@@ -60,6 +63,7 @@ func NewServer(config *configuration.Configuration) (*Server, error) {
ProtobufServer: protobufServer,
ClusterConfig: clusterConfig,
HttpApi: httpApi,
+ GraphiteApi: graphiteApi,
Coordinator: coord,
AdminServer: adminServer,
Config: config,
@@ -93,6 +97,14 @@ func (self *Server) ListenAndServe() error {
log.Info("Starting admin interface on port %d", self.Config.AdminHttpPort)
go self.AdminServer.ListenAndServe()
+ if self.Config.GraphiteEnabled {
+ if self.Config.GraphitePort <= 0 || self.Config.GraphiteDatabase == "" {
+ log.Warn("Cannot start graphite server. please check your configuration")
+ } else {
+ log.Info("Starting Graphite Listener on port %d", self.Config.GraphitePort)
+ go self.GraphiteApi.ListenAndServe()
+ }
+ }
log.Info("Starting Http Api server on port %d", self.Config.ApiHttpPort)
return nil
Something went wrong with that request. Please try again.