New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge listener and replay functionality #46

Merged
merged 20 commits into from Oct 31, 2013

Conversation

Projects
None yet
3 participants
@buger
Owner

buger commented Oct 27, 2013

I like architecture of http://fluentd.org/, it have multiple input and output sources, and fluentd instances can be combined like LEGO bricks.

I want same for Gor, and think It should be logical move, and it will improve overall flexibility.

Examples of current functionality:

Listen data raw socket and replay it different Gor instance

gor --input-raw 9000 --output-tcp replay.server:27017
gor --input-tcp 27017 --output-http staging.dev

File replay:

gor --input-raw 9000 --output-file requests.gor # capture
gor --input-file requests.gor --output-http staging.org # replay

Replay to multiple hosts:

gor --input-raw 9000 --output-tcp replay.server:27017
gor --input-tcp 27017 --output-http staging.org --output-http "dev.org|10"

Examples of functionality that will be possible after change:

Listen and replay with one Gor instance:

gor --input-raw 9000 --output-http "staging.org"

HTTP Input #45:

gor --input-http 8080 --output-tcp "replay.org:27017"

And much more.

What would you think @slawosz @dcarley ?

@slawosz

This comment has been minimized.

Show comment
Hide comment
@slawosz

slawosz Oct 21, 2013

Collaborator

I like the idea and new, clean flag names.

Collaborator

slawosz commented Oct 21, 2013

I like the idea and new, clean flag names.

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 21, 2013

Owner

This is also allow to bring much more Go-like architecture with simple modules and interfaces.

Owner

buger commented Oct 21, 2013

This is also allow to bring much more Go-like architecture with simple modules and interfaces.

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 21, 2013

Owner

Here is basic architecture diagram:

image

Owner

buger commented Oct 21, 2013

Here is basic architecture diagram:

image

@dcarley

This comment has been minimized.

Show comment
Hide comment
@dcarley

dcarley Oct 21, 2013

Collaborator

👍 from me.

Collaborator

dcarley commented Oct 21, 2013

👍 from me.

@buger buger referenced this pull request Oct 23, 2013

Closed

Introduce config file #50

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 25, 2013

Owner

@slawosz @dcarley pls check this branch, concept is working, and it's more awesome than i thought :)

https://github.com/buger/gor/tree/modules_refactoring

You can start by playing like this:

gor --input-dummie 0 --output-dummie 0

And continue with smth like:

gor --input-raw :27017 --output-http localhost:27018
Owner

buger commented Oct 25, 2013

@slawosz @dcarley pls check this branch, concept is working, and it's more awesome than i thought :)

https://github.com/buger/gor/tree/modules_refactoring

You can start by playing like this:

gor --input-dummie 0 --output-dummie 0

And continue with smth like:

gor --input-raw :27017 --output-http localhost:27018
@slawosz

This comment has been minimized.

Show comment
Hide comment
@slawosz

slawosz Oct 25, 2013

Collaborator

So, after 2 minutes of looking (no more time now :() here is core of gor now?: https://github.com/buger/gor/blob/modules_refactoring/emitter.go#L20

Each type of i/o implements io.{Read,Write} and uses bytes?

Collaborator

slawosz commented Oct 25, 2013

So, after 2 minutes of looking (no more time now :() here is core of gor now?: https://github.com/buger/gor/blob/modules_refactoring/emitter.go#L20

Each type of i/o implements io.{Read,Write} and uses bytes?

@dcarley

This comment has been minimized.

Show comment
Hide comment
@dcarley

dcarley Oct 25, 2013

Collaborator

Wow, I like the look of it. Will have a play around.

Collaborator

dcarley commented Oct 25, 2013

Wow, I like the look of it. Will have a play around.

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 25, 2013

Owner

@slawosz basically yes. Input and Output just satisfy standard io.Reader and io.Writer interfaces, which gives you ability to use any standard library (bufio ?).

Owner

buger commented Oct 25, 2013

@slawosz basically yes. Input and Output just satisfy standard io.Reader and io.Writer interfaces, which gives you ability to use any standard library (bufio ?).

@slawosz

This comment has been minimized.

Show comment
Hide comment
@slawosz

slawosz Oct 25, 2013

Collaborator

It is beautiful piece of code 👍

Collaborator

slawosz commented Oct 25, 2013

It is beautiful piece of code 👍

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 27, 2013

Owner

Converted issue to pull-request

Owner

buger commented Oct 27, 2013

Converted issue to pull-request

func (i *RAWInput) Read(data []byte) (int, error) {
buf := <-i.data
copy(data, buf)

This comment has been minimized.

@slawosz

slawosz Oct 27, 2013

Collaborator

Why we have copy here?

@slawosz

slawosz Oct 27, 2013

Collaborator

Why we have copy here?

This comment has been minimized.

@buger

buger Oct 27, 2013

Owner

Maybe it's not needed here. Just want to make sure that data will be immutable.

@buger

buger Oct 27, 2013

Owner

Maybe it's not needed here. Just want to make sure that data will be immutable.

This comment has been minimized.

@buger

buger Oct 27, 2013

Owner

Oh, btw, i remembered. It needed :)

Thing is that io.Reader interface says that Read function should received slice as argument and we should write to it.

If i just do data = buf it will rewrite original data pointer, and original data still be empty. Thats why we use copy.

@buger

buger Oct 27, 2013

Owner

Oh, btw, i remembered. It needed :)

Thing is that io.Reader interface says that Read function should received slice as argument and we should write to it.

If i just do data = buf it will rewrite original data pointer, and original data still be empty. Thats why we use copy.

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 27, 2013

Owner

This pull-request is long-going, and require adding lot of tests. Also it not support rate-limiting.

Owner

buger commented Oct 27, 2013

This pull-request is long-going, and require adding lot of tests. Also it not support rate-limiting.

@slawosz

This comment has been minimized.

Show comment
Hide comment
@slawosz

slawosz Oct 27, 2013

Collaborator

Oh, testing right now will be much easier :)

Collaborator

slawosz commented Oct 27, 2013

Oh, testing right now will be much easier :)

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 29, 2013

Owner

@slawosz @dcarley I added tests and missing plugins. Now this branch should support all features of previous versions (except some minor stuff like custom headers and elastic search stats, but this will be added soon). If you have time pls test it. Thanks!

Owner

buger commented Oct 29, 2013

@slawosz @dcarley I added tests and missing plugins. Now this branch should support all features of previous versions (except some minor stuff like custom headers and elastic search stats, but this will be added soon). If you have time pls test it. Thanks!

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 30, 2013

Owner

Added custom headers and ElasticSearch plugin. Also added option to round-robin traffic if have multiple outputs:

--split-output=true
Owner

buger commented Oct 30, 2013

Added custom headers and ElasticSearch plugin. Also added option to round-robin traffic if have multiple outputs:

--split-output=true

buger added a commit that referenced this pull request Oct 31, 2013

Merge pull request #46 from buger/module_architecture
Merge listener and replay functionality

@buger buger merged commit ccf6723 into master Oct 31, 2013

@buger

This comment has been minimized.

Show comment
Hide comment
@buger

buger Oct 31, 2013

Owner

MERGED!

Owner

buger commented Oct 31, 2013

MERGED!

@dcarley

This comment has been minimized.

Show comment
Hide comment
@dcarley

dcarley Oct 31, 2013

Collaborator

Awesome work 🍻

Collaborator

dcarley commented Oct 31, 2013

Awesome work 🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment