LVM module #184

avnik · 2016-08-22T17:52:33Z

This is straight rewrite of mantl-storage-setup.py (at the moment)

Architecture caveats:
1 we need to install lvm tools, and start lvm-metad service before we start
and we can't issue install requirement in lvm module.
2 we need to express all LVM structure in single hcl expression, otherwise we
can't plan/check VG/LV part, until PV/VG part commited.

Possible future ways:

all storage/lvm structure as single expression
step-by-step executing to eliminate difference between "desired state" (by hcl expression) and "current state" (which exposed by lvs/pvs/vgs commands)

Funny go language issues:
we can exec.Command(name, args...) but not `exec.Command(name, arg, more_args...)``

So let discuss it

avnik · 2016-08-22T17:53:40Z

suggestions how to make LVM related code testable also welcomed

rebeccaskinner · 2016-08-23T17:51:59Z

resource/lvm/vg/util.go

+		}
+	}
+	return "", fmt.Errorf("too many symlinks for %s", dev)
+}


Could we use filepath.EvalSymlinks here?

https://golang.org/pkg/path/filepath/#EvalSymlinks

definelly yes

rebeccaskinner · 2016-08-23T17:54:52Z

Could we consider using LogicalVolume, PhysicalVolume and VolumeGroup instead of lv, pv and vg? I know that's more verbose than the tool itself but might make scripts more readable for people who aren't as familiar with lvm?

stevendborrelli · 2016-10-06T20:30:59Z

please make sure your example has a check for the pvs, etc. commands or it will fail

stevendborrelli · 2016-10-06T20:32:56Z

Errors are not clear during a failure. You are logging tasks in INFO but it is not clear why this is failing:

root/lvm.lv.lv-test:
    Messages:
    Has Changes: no
    Changes: No changes

root/lvm.fs.mnt-me:
    Error: exit status 2
    Messages:
    Has Changes: no
    Changes: No changes

root/lvm.vg.mantl:
    Messages:
    Has Changes: no
    Changes: No changes

Summary: 2 errors, 0 changes

 * root: error in dependency "root/lvm.fs.mnt-me"
 * root/lvm.fs.mnt-me: exit status 2

stevendborrelli · 2016-10-06T20:34:29Z

examples/lvm/lvm.hcl

+}
+
+lvm.fs "mnt-me"  {
+  device = "/dev/mapper/mantl-test"


this should be /dev/mapper/lv-test

stevendborrelli · 2016-10-06T20:47:31Z

The module should check for the lvm commands (pvs, lvs, .etc.) before trying to execute.
errors are not propagated back to the user
I think some of the checks are not working. I am getting a clear on the /mnt/ filesystem even though it is not mounted. Perhaps the checks needs to go a little deeper and confirm the right configuration.

We really need to throw a lot of errors at it and make sure it is providing helpful feedback.

stevendborrelli · 2016-10-06T21:09:58Z

Are pvs created automatically? I think it would be helpful to separate that out as a module

root/lvm.vg.mantl:
    Messages:
    Has Changes: yes
    Changes:
        /dev/sdb: "<none>" => "member of volume group mantl"
        mantl: "<not exists>" => "/dev/sdb"

Not really clear what is happening here. Is the lv being created or the vg?

root/lvm.lv.lv-test:
    Messages:
    Has Changes: yes
    Changes:
        group: mantl: "<not exists>" => "created"

arichardet · 2016-10-14T20:08:31Z

@avnik You have failing tests - please take a look and make appropriate changes. Thanks!

arichardet · 2016-10-18T15:33:34Z

examples/lvm/lvm.hcl

@@ -0,0 +1,18 @@
+lvm.vg "mantl" {


Please remove "mantl" in all places and replace with "test" or something else.

done (also reoved from history)

arichardet · 2016-10-18T15:38:44Z

examples/lvm/lvm.hcl

@@ -0,0 +1,18 @@
+lvm.vg "mantl" {
+  name = "mantl"
+  devices = [ "/dev/sdb" ]


can you change this to a parameter so users can provide the value; you could set "/dev/sdb" as default.

arichardet · 2016-10-18T15:45:40Z

resource/lvm/fs/fs.go

+)
+
+type ResourceFS struct {
+	mount           *Mount


mount, lvm, etc. should be capitalized since the struct is exported

arichardet · 2016-10-18T15:51:30Z

resource/lvm/fs/preparer.go

+	device, err := render.Render("device", p.Device)
+	if err != nil {
+		return nil, err
+	}


This is no longer necessary and is handled for you. You can simply set "What" to p.device in Mount, etc.

arichardet · 2016-10-18T15:53:05Z

resource/lvm/lv/preparer.go

+	group, err := render.Render("group", p.Group)
+	if err != nil {
+		return nil, err
+	}


These checks are now done for you; just use the value of p.Group, etc.

arichardet · 2016-10-18T20:20:31Z

resource/lvm/lowlevel/exec.go

+func (*OsExec) Run(prog string, args []string) error {
+	log.WithField("module", "lvm").Infof("Executing %s: %v", prog, args)
+	e := exec.Command(prog, args...).Run()
+	if e == nil {


Probably only need the err != nil case here.

arichardet · 2016-10-18T20:22:25Z

resource/lvm/lowlevel/exec_test.go

+	"testing"
+)
+
+func TestExecRun(t *testing.T) {


It might be easier to follow if you have one Test[FuncName], and then use t.Run to test the different cases of success and failure resulting in error. Using t.Run is what we are trying to do now, so updating throughout is probably a good idea.

arichardet · 2016-10-18T20:30:26Z

resource/lvm/lv/lv.go

+)
+
+type ResourceLV struct {
+	group      string


group, name, etc. should be capitalized since this struct is exported.

arichardet · 2016-10-18T20:31:12Z

resource/lvm/lv/lv.go

+	status := &Status{}
+	ok, err := r.checkVG()
+	if err != nil {
+		return nil, err


Probably a good idea to wrap the errors with "check", "apply", etc. to give the user more information.

arichardet · 2016-10-18T20:33:04Z

resource/lvm/vg/vg.go

+
+	pvs, err := r.lvm.QueryPhysicalVolumes()
+	if err != nil {
+		return nil, err


Wrapping errors here in check and apply would be helpful for the user.

CodeClimate aggroing on then

Also move init() to preparer.go as in all other resources

coverage: 98.2% of statements

LVM module

BrianHicks added the in progress label Aug 22, 2016

rebeccaskinner reviewed Aug 23, 2016
View reviewed changes

avnik force-pushed the feature/lvm branch 2 times, most recently from ced335a to 8768f24 Compare September 15, 2016 13:28

stevendborrelli added this to the 0.3 milestone Sep 17, 2016

BrianHicks removed the in progress label Sep 19, 2016

avnik force-pushed the feature/lvm branch from 8768f24 to 4e05de7 Compare September 26, 2016 14:59

avnik force-pushed the feature/lvm branch 2 times, most recently from 1911304 to af116b9 Compare October 6, 2016 18:24

stevendborrelli added the module label Oct 6, 2016

stevendborrelli changed the title ~~[WIP] LVM proof-of-concept~~ LVM module Oct 6, 2016

stevendborrelli reviewed Oct 6, 2016

View reviewed changes

examples/lvm/lvm.hcl

}

lvm.fs "mnt-me" {

device = "/dev/mapper/mantl-test"

Copy link

Member

stevendborrelli Oct 6, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be /dev/mapper/lv-test

avnik force-pushed the feature/lvm branch from 095298b to c00ba39 Compare October 14, 2016 17:44

stevendborrelli modified the milestones: 0.4.0, 0.3.0 Oct 14, 2016

avnik force-pushed the feature/lvm branch from 96bbe8c to 4b4975a Compare October 18, 2016 11:21

arichardet requested changes Oct 18, 2016

View reviewed changes

avnik force-pushed the feature/lvm branch 5 times, most recently from 5ecc8c0 to 092c008 Compare October 24, 2016 19:58

avnik and others added 19 commits November 10, 2016 11:28

lvm: remove some unused code

41108a8

lvm: demote device existence check to low-level

d7e7c18

lvm: whitespace in comments

e754c1d

CodeClimate aggroing on then

lvm: missing comments for vg_test

baa0b04

lvm: fix failing test, and reenable it

11add34

lvm/vg: Documet preparer, add remove option

0b57634

lvm/lv: LV preparer documentation

dfa61fc

lvm/fs: Document preparer

a8d27b2

lvm: rename resources to full names

1377e85

Also move init() to preparer.go as in all other resources

lvm/vg: More tests

1c813cf

coverage: 98.2% of statements

lvm: avoid using braces for variable scoping

caca3ec

lvm: more tests

acc71b2

lvm: enable documentation

0e4665a

lvm: update to use new api with context.Context

d1e2d57

lvm: more error wrapping

7f592d0

lvm: fix CC warnings

23feae8

fix preparers

f2ef449

add preparer tests

8a35f18

mock symlink evaluation

19c98dc

rebeccaskinner force-pushed the feature/lvm branch from 0b02675 to 19c98dc Compare November 10, 2016 17:28

Rebecca Skinner added 5 commits November 10, 2016 11:35

comment test funcs for codeclimate

d97a1df

use nil instead of empty string or 0 for retry

6c61d44

name context in preparers

11be508

prune dead code

59f7bef

linter fixes

564500c

arichardet approved these changes Nov 10, 2016

View reviewed changes

arichardet merged commit 11beb43 into master Nov 10, 2016

arichardet deleted the feature/lvm branch November 10, 2016 20:17

arichardet mentioned this pull request Nov 14, 2016

LVM module #270

Closed

BrianHicks pushed a commit that referenced this pull request Dec 22, 2016

Merge pull request #184 from asteris-llc/feature/lvm

35b57d7

LVM module

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LVM module #184

LVM module #184

avnik commented Aug 22, 2016

avnik commented Aug 22, 2016

rebeccaskinner Aug 23, 2016

avnik Aug 23, 2016

rebeccaskinner commented Aug 23, 2016

stevendborrelli commented Oct 6, 2016

stevendborrelli commented Oct 6, 2016

stevendborrelli Oct 6, 2016

stevendborrelli commented Oct 6, 2016 •

edited

stevendborrelli commented Oct 6, 2016

arichardet commented Oct 14, 2016

arichardet Oct 18, 2016

avnik Oct 19, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

arichardet Oct 18, 2016

LVM module #184

LVM module #184

Conversation

avnik commented Aug 22, 2016

avnik commented Aug 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rebeccaskinner commented Aug 23, 2016

stevendborrelli commented Oct 6, 2016

stevendborrelli commented Oct 6, 2016

Choose a reason for hiding this comment

stevendborrelli commented Oct 6, 2016 • edited

stevendborrelli commented Oct 6, 2016

arichardet commented Oct 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevendborrelli commented Oct 6, 2016 •

edited