Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[simulator][machine] Simulator config, command and improve logic #13

Merged
merged 18 commits into from
Nov 23, 2016

Conversation

at15
Copy link
Member

@at15 at15 commented Nov 17, 2016

  • logging library. Logrus 7ecdba3
  • config simulator using yaml and command line. (might use viper) (we only consider machine for now)
    • machine numbers
    • start, end, interval
  • change logic for how simulator generate data, currently, when machine is added to the simulator, it's just their series been added, the MachineSimulator does not know about machine, as suggested by @czheo in [WIP] Data generation #9 the simulator should be aware of machine. I am not very clear how I should change the logic but the output (which is the input for serializer) should be the same, since we are still using the simplest and most verbose format, one series + one data point instead of one series + multiple data points when generating data. When loading the data, it really depends on how the database provide interface for bulk loading, batches can be done in loader, so this should not be the concern for simulator, using database independent format can allow generated be used on multiple databases, which is useful if the data generation is resource consuming.
  • write to file
  • deserialize file (the part is more used in loader, but it should also be used in serializer's test) leave it it to loader

There are a lot more that could be configured for the machine, like we can add higher level things like rack, data center, for micro benchmark, they are just tags, for real bechmark, those information can be used for deploy probes to machines in different rack, dc. However, they are just too much for now.

- alias sim
- No real logic is added
@at15 at15 added this to the 0.0.1 milestone Nov 17, 2016
- use cobra to bind command line flag
- use logrus for logging
which introduced a lot of packages
@at15
Copy link
Member Author

at15 commented Nov 21, 2016

An example snippet of machine simulator config

simulator:
    machine:
        num: 100
        # start time in ISO 8601
        start: 1997-07-16T19:20:30.45+01:00
        # end time in ISO 8601x
        end: 1997-07-16T19:20:31.45+01:00
        # in second
        step: 10
  • use viper to get proper time object 24e067d
  • generate simulator from config

- Add machine simulator config struct, which supports: num, start, end, step
- add util function for read config file in test (TODO: don't know if it will have sideffect if is called multiple times)
- the machine.go is already 150+ lines with current dumbest simulation logic,
so should put it in sub package and split to several files
- Add format script in Ayi, which will call gofmt
- machine is the pojo and factory for machine
- simulator is the real machine simulator
@at15
Copy link
Member Author

at15 commented Nov 21, 2016

The plan for refactor the generator logic is like the following

  • *rename generator package to distribution
  • remove ConstantIntGenerator since it contains both time and value, which is an unnecessary abstraction a908a4a
  • remove the SeriesWithGenerator since int generator is removed a908a4a
  • add time and value package with FixedInterval and FixedValue to replace the old constant int generator

And the machine simulator logic used to be

  • AddMachine
  • AddSeriesWithGenerator
  • Loop and generate value by Series

Now it will be

  • AddMachine
  • Loop machine and generate a group of series, the group:
    • share one time stamp from FixedInterval
    • generate different value from FixedValue

- intial time and value sub package under generator
- add fixed interval
- add constant
- TestNewGenerator shows it is working
- add functions ending with New, in order to have old code as example, will remove old and change name after migration
- add value generator interface
- when generate default machine, add seriesWithValueGenerater into machine
- machine has different array for keep int and double series
- see TestMachineSimulatorNew
- TODO: why I saw more cpu_core=3 in log ouput
- TODO: remove old generator code
- generate machine
- add default machine
- start
- related property and methods in simulator interface
- delete interface for point generator
- delete constant int point generator
- delete series with point generator
@at15
Copy link
Member Author

at15 commented Nov 23, 2016

The plan for the simulator command is like the following

  • create simulator config based on command line arguments (may support config in the future)
  • create machine simulator config based on simulator config
  • create machine simulator based on machine simulator config
  • start the simulator
  • clean up

some changes need to be made

  • add \n for the json serializer (or maybe add it to the writer? I prefer to put it in the serializer)
  • support output to file

- the simulator config struct is actually never used
- TODO: not output in current default setting, might be problem with config file
It seems the serializer is not properly initialized
````
DEBU[0000] enable debug logging                          pkg=x.util
DEBU[0000] config file xephon-b.yml is loaded            pkg=x.cmd
DEBU[0000] triggered simulator command                   pkg=x.cmd
DEBU[0000] machine simulator start                       pkg=x.s.machine
DEBU[0000] need to serialize                             pkg=x.s.machine
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x498eed]

goroutine 21 [running]:
panic(0x7bc3e0, 0xc420016120)
	/home/at15/app/go/src/runtime/panic.go:500 +0x1a1
github.com/xephonhq/xephon-b/pkg/simulator/machine.(*MachineSimulator).Start.func2(0xc42007cde0, 0xc4200aa630, 0xc42007ce40, 0xc42017c290)
	/home/at15/workspace/src/github.com/xephonhq/xephon-b/pkg/simulator/machine/simulator.go:84 +0x11d
created by github.com/xephonhq/xephon-b/pkg/simulator/machine.(*MachineSimulator).Start
	/home/at15/workspace/src/github.com/xephonhq/xephon-b/pkg/simulator/machine/simulator.go:92 +0x20e
````
- treated output as encoding and cased the null pointer problem
- change example num to 3 for less output when try the command
- run `xb sim --debug --type json` to see json output
- TODO: json serialize should have `\n` in order to be read by others
- ignore *.dat file as its the default output file suffix
- TODO: use append may not be the most efficient way. But adding a wrapper for writer might be more messy
since the writer should know little(nothing) about the serializer
- though it seems file without calling f.Close
@at15
Copy link
Member Author

at15 commented Nov 23, 2016

now a small problem for json ouput, it has an extra empty line

{"v":1,"t":869077270450000000,"name":"machine","tag":{"cpu_core":"3","host":"default-2","os":"ubuntu16.04"}}
{"v":1,"t":869077270450000000,"name":"machine","tag":{"cpu_core":"3","host":"default-2","os":"ubuntu16.04"}}

I think the reader could have check when deserialize the data, so I will just ignore it. PTAL @czheo I will merge it tomorrow if no response from u.

@at15
Copy link
Member Author

at15 commented Nov 23, 2016

LGTM
merge without review. 没时间了,快上车

@at15 at15 changed the title [WIP][gen] Simulator config, command and improve logic [simulator][machine] Simulator config, command and improve logic Nov 23, 2016
@at15 at15 merged commit 86b47e8 into master Nov 23, 2016
@at15 at15 removed the working label Nov 23, 2016
@at15 at15 mentioned this pull request Nov 23, 2016
@at15 at15 mentioned this pull request Dec 15, 2016
4 tasks
@at15 at15 deleted the feature/data-generation branch March 12, 2018 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants