Add persistent storage #460

cdrage · 2015-12-15T19:47:16Z

DONE:

~~Initial work on persistent storage. Work's correctly. Just need to clean-up most of the code.~~

~~It's a Work-In-Progress so excuse the multiple log messages, but you get the gist of how I implemented it.~~

~~Code comments / docs~~
~~More tests!~~

Name                        Stmts   Miss  Cover   Missing
---------------------------------------------------------
atomicapp/requirements.py      36      3    92%   40, 43, 46
----------------------------------------------------------------------
Ran 5 tests in 0.009s

tests cover 92% and under kubernetes.py persistent_storage function too

~~Clean up log messages~~

TODO:

Grep kubectl pvc output or something to see if there is actually any persistent volumes yet for Kubernetes. Warn the user if so.
Logs are a mess (this will be cleaned up in refactor logging #134)
Documentation in Nulecule spec or nulecule-library example should be added.
Split into different commits

Notes:

For now this will only be for Kubernetes persistent storage. OpenShift will be added at a future date (Docker maybe too with the Flocker plugin).
Only creating persistent storage has been adding, no removing (yet?) we need to discuss this.

cdrage · 2015-12-18T18:18:10Z

atomicapp/providers/kubernetes.py

+            cmd = [self.kubectl, "create", "-f", tmp_file, "--namespace=%s" % self.namespace]
+            if self.config_file:
+                cmd.append("--kubeconfig=%s" % self.config_file)
+            self._call(cmd)


should we add deleting / removing persistent storage or just running?

I guess for now we can stick with this and revisit when we do uninstall.

ping @dustymabe how should we check before running this? another if statement at the beginning of the function to check what actions are available?

Right.. I would move "checks" to the beginning of the function. No reason to do work if we fail the checks

Given this is "persistent" we probably don't want to ever be deleting. I believe OpenShift does not delete storage when all objects are destroyed. It's treated special so as to do no unexpected harm.

cdrage · 2015-12-18T18:24:30Z

Oh, another note, logging is starting to get SO MESSY. Way too much logging coming out now when running each application, this will be fixed in #134

dustymabe · 2015-12-18T19:48:29Z

Ok.. ran through it once.. not sure I quite understand it all just yet.. still digesting. I think this would be better/easier to understand if you broke up the big commit into smaller commits with good commit messages. Let's chat later today or next week about the architecture

cdrage · 2015-12-18T20:40:27Z

atomicapp/nulecule/base.py

+        # Pass configuration, path of the app, graph, provider as well as dry-run for provider init()
+        if REQUIREMENTS_KEY in self.graph[0]:
+            logger.debug("Requirements key detected. Running action.")
+            Requirements(self.config, self.basepath, self.graph[0][REQUIREMENTS_KEY], provider_key, dryrun).run()


should be

requirements = Requirements(etc...) requirements.run()

cdrage · 2015-12-18T22:38:23Z

Merged in changes from @dustymabe 's suggestions :) Let's split this into different commit's and have a look at this after the weekend

cdrage · 2015-12-21T17:00:03Z

Okay, so I've split the commits as well as added checking kubectl get pv to see if any persistent volumes actually exist in order to claim them (with persistentVolumeClaim).

Ping @dustymabe

The only thing that's left is documentation to be added.

We will need a discussion with @bexelbie about this. With a new feature like this added, it's hard for users to see the significant changes to atomicapp if we only update the Nulecule spec. A detailed documentation that lists all Nulecule functions in atomicapp should be here. I can volunteer to help add this.

Side note:
Let's add OpenShift functionality after this PR once we know that Kubernetes 100% works. (and possibly Docker with the flocker plugin? maybe. not many people using it at the moment IMO)

dustymabe · 2015-12-22T14:07:45Z

atomicapp/nulecule/base.py

+        # Pass configuration, path of the app, graph, provider as well as dry-run for provider init()
+        if REQUIREMENTS_KEY in self.graph[0]:
+            logger.debug("Requirements key detected. Running action.")
+            r = Requirements(self.config, self.basepath, self.graph[0][REQUIREMENTS_KEY], provider_key, dryrun)


Split up this line.. too long

cdrage · 2015-12-22T15:48:15Z

atomicapp/providers/kubernetes.py

+                "%s action is not available for provider %s. Doing nothing." %
+                (action, self.key))
+            return
+


@dustymabe like this? don't see any other way since each cmd call will be different for delete / create / etc. when more functions are added.

Would rather do:

if action not in ['run']: logger.warning()

In the future when we add more it would be:

if action not in ['run', 'stop', etc..]: logger.warning()

cdrage · 2015-12-22T15:52:16Z

Tested with changes and can confirm functionality:

dropbox/dev/atomicapp  persistent-storage ✗                                                                                                                                                                                                        22h56m ◒  ⍉
▶ kubectl get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM     REASON

dropbox/dev/atomicapp  persistent-storage ✗                                                                                                                                                                                                         22h56m ◒  
▶ kubectl get pvc
NAME                   LABELS    STATUS    VOLUME
var-lib-mongodb-data   map[]     Pending   
var-log-mongodb        map[]     Pending   

dropbox/dev/atomicapp  persistent-storage ✗                                                                                                                                                                                                         22h56m ◒  
▶ kubectl get po
NAME                   READY     STATUS                                                RESTARTS   AGE
helloapache            0/1       Image: centos/httpd is ready, container is creating   0          1m
k8s-master-127.0.0.1   3/3       Running                                               5          1m

cdrage · 2015-12-22T16:18:49Z

atomicapp/requirements.py

+        p = plugin.getProvider(provider)
+        self.provider = p(config, basepath, dryrun)
+        self.provider.init()
+


dustymabe · 2015-12-22T16:30:09Z

@goern @aweiteka. we would really like to get your input on this before we merge. If you are available today or tomorrow before the holidays that would be great.

dustymabe · 2015-12-22T16:32:38Z

atomicapp/requirements.py

+from atomicapp.plugin import Plugin
+
+plugin = Plugin()
+plugin.load_plugins()


move to inside of __init__

aweiteka · 2015-12-22T19:40:43Z

Sorry I haven't been following this from the start. Question: where is the volume supposed to be mounted so the application can actually use it? In my experience if persistent storage is needed it's defined in a template and referenced in the pod spec so it's actually mounted. But to specify all of that in Nulecule feels like we're duplicating kubernetes.

I think the spec is lacking here but this looks like a good start to implement what we have. LGTM

goern · 2015-12-27T19:51:15Z

@aweiteka I think you got the approach right: the Nulecule Specification is making a statement about what type of storage an application needs, Atomic App is going to auto generate a Kubernetes/OpenShift snippet for that and the application can use it. We can also keep the storage requirements within the artifacts of the provider, but than we will obfuscate the application, eg the deployment guy will not be able to parameterize the size or type of the storage.

Furthermore, what is the spec missing wrt a storage requirement?

dustymabe · 2016-01-06T03:55:54Z

considering @aweiteka's new comment in projectatomic/nulecule#193

the application artefact will define how it wants to use the volume, where it should be mounted. We don't need to provide a path in the spec.

I think we should move forward with this review. @aweiteka @goern can you please review so that we can get this merged?

aweiteka · 2016-01-06T14:00:38Z

atomicapp/providers/kubernetes.py

+                cmd.append("--kubeconfig=%s" % self.config_file)
+            lines = self._call(cmd)
+
+            # If there are no persistent volumes to claim, warn the user


There are more error cases. I can think of two:

claim with that name has already been created.

user is claiming more space than allowed.

These (and other errors like this I can't think of) can be generally caught with one check, I would think.

Ah yes, this should error out automatically though when calling the API. But it would be nice to parse and create a good logging output for this :)

aweiteka · 2016-01-06T14:06:06Z

Nice work! LGTM.

cdrage · 2016-01-06T15:17:29Z

^^ changed the small comment in code from @vpavlin 's comment :) I'll update notes / comments in a different PR for changes unrelated to persistent storage.

dustymabe · 2016-01-06T15:54:58Z

@cdrage, since there has been a lot of work in master can we get a rebase on it. Might be good to test on top of master anyway.

cdrage · 2016-01-06T18:57:27Z

Rebased and checks pass!

dustymabe · 2016-01-07T15:14:34Z

With @aweiteka LGTM and a LGTM from me.. merging.

Add persistent storage for kubernetes

cdrage force-pushed the persistent-storage branch 6 times, most recently from 93140ad to a0073c7 Compare December 18, 2015 18:16

cdrage reviewed Dec 18, 2015
View reviewed changes

cdrage force-pushed the persistent-storage branch from a0073c7 to e6355e3 Compare December 18, 2015 18:20

cdrage added the done label Dec 18, 2015

cdrage changed the title ~~[WIP] Add persistent storage~~ Add persistent storage Dec 18, 2015

cdrage reviewed Dec 18, 2015
View reviewed changes

cdrage force-pushed the persistent-storage branch from e6355e3 to 0ed429f Compare December 18, 2015 22:25

cdrage force-pushed the persistent-storage branch 2 times, most recently from 806a5ba to 1aaa682 Compare December 21, 2015 16:55

cdrage added the need_doc label Dec 21, 2015

dustymabe reviewed Dec 22, 2015
View reviewed changes

cdrage force-pushed the persistent-storage branch 2 times, most recently from 117e7cd to b0970fc Compare December 22, 2015 15:47

cdrage reviewed Dec 22, 2015
View reviewed changes

cdrage force-pushed the persistent-storage branch 2 times, most recently from 377c149 to ea722d5 Compare December 22, 2015 16:17

cdrage reviewed Dec 22, 2015
View reviewed changes

atomicapp/requirements.py

p = plugin.getProvider(provider)

self.provider = p(config, basepath, dryrun)

self.provider.init()

Copy link

Member Author

cdrage Dec 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^^

cdrage force-pushed the persistent-storage branch from ea722d5 to c703374 Compare December 22, 2015 16:32

dustymabe reviewed Dec 22, 2015
View reviewed changes

aweiteka mentioned this pull request Dec 23, 2015

Persistent storage types projectatomic/nulecule#193

Open

aweiteka reviewed Jan 6, 2016
View reviewed changes

cdrage force-pushed the persistent-storage branch from c703374 to 4986521 Compare January 6, 2016 15:16

cdrage added 4 commits January 6, 2016 13:55

Add persistent storage core

492a73e

Add Kubernetes persistent storage functionality

fdec359

Test requirements.py persistent storage

6970e84

Warn if no persistent volumes exist to claim

58aaa15

cdrage force-pushed the persistent-storage branch from 4986521 to 58aaa15 Compare January 6, 2016 18:57

dustymabe added a commit that referenced this pull request Jan 7, 2016

Merge pull request #460 from cdrage/persistent-storage

c2ec7fb

Add persistent storage for kubernetes

dustymabe merged commit c2ec7fb into projectatomic:master Jan 7, 2016

dustymabe mentioned this pull request Jan 7, 2016

We need an implementation of Nulecule's Storage Requirements #62

Closed

cdrage mentioned this pull request Jan 7, 2016

Updated docs / list of current functions from Nulecule specification #471

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add persistent storage #460

Add persistent storage #460

cdrage commented Dec 15, 2015

cdrage Dec 18, 2015

dustymabe Dec 22, 2015

cdrage Dec 22, 2015

dustymabe Dec 22, 2015

aweiteka Jan 6, 2016

cdrage commented Dec 18, 2015

dustymabe commented Dec 18, 2015

cdrage Dec 18, 2015

cdrage commented Dec 18, 2015

cdrage commented Dec 21, 2015

dustymabe Dec 22, 2015

cdrage Dec 22, 2015

dustymabe Dec 22, 2015

cdrage Dec 22, 2015

cdrage commented Dec 22, 2015

cdrage Dec 22, 2015

dustymabe commented Dec 22, 2015

dustymabe Dec 22, 2015

aweiteka commented Dec 22, 2015

goern commented Dec 27, 2015

dustymabe commented Jan 6, 2016

aweiteka Jan 6, 2016

cdrage Jan 6, 2016

aweiteka commented Jan 6, 2016

cdrage commented Jan 6, 2016

dustymabe commented Jan 6, 2016

cdrage commented Jan 6, 2016

dustymabe commented Jan 7, 2016

Add persistent storage #460

Add persistent storage #460

Conversation

cdrage commented Dec 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdrage commented Dec 18, 2015

dustymabe commented Dec 18, 2015

Choose a reason for hiding this comment

cdrage commented Dec 18, 2015

cdrage commented Dec 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdrage commented Dec 22, 2015

Choose a reason for hiding this comment

dustymabe commented Dec 22, 2015

Choose a reason for hiding this comment

aweiteka commented Dec 22, 2015

goern commented Dec 27, 2015

dustymabe commented Jan 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aweiteka commented Jan 6, 2016

cdrage commented Jan 6, 2016

dustymabe commented Jan 6, 2016

cdrage commented Jan 6, 2016

dustymabe commented Jan 7, 2016