[Heron-3723] Add support for Persistent Volumes for stateful storage #3725

…Map name.

…ding to Stateful Set.

…gMap.

Adding checks for null pointers. Default constructed V1 objects tend to have uninitialised fields set to null by default. Extracting <getConfigMaps> to method to support mocking.

…> from <V1PodTemplate>.

Judging from <release-11.0.0/kubernetes/src/main/java/io/kubernetes/client/openapi/apis/CoreV1Api.java> "optional" means the field can be set to <null>.

…mplate names.

…d Template names.

…emplate names.

…Pod Template names.

…oyed Pod Template.

…eLocation>.

TODO: get API key for K8s.

Removed illegal reflection access to avoid support issues with newer testing frameworks.

RBAC must be configured using Role/ClusterRole and RoleBinding/ClusterRoleBinding to the Heron <heron-apiserver> ServiceAccount.

Heron should have the final say on the Pod Spec. This is as much a point of security as an operational one.

…FromTemplate>.

…tainer.

Environment variables merged with Heron defaults taking precedence.

Resource Limits merged with Heron defaults taking precedence.

… to submit.

Updated RBAC API version and added a commented flag command to disable to Pod Templates.

Refactored <getContainer> to <configureExecutorContainer>. Permitting additional containers for side-car purposes.

Switched to <TreeSet> with custom comparator for <V1EnvVar> name. <V1EnvVar>'s comparator performs a complete element wise comparison.

Merged executor container ports with ports provided in Pod Template. Heron defaults take precedence.

Merge volume mounts with those provided in Pod Template. Heron defaults take precedence.

general cleanup of new code and comments.

Error check for malformed Pod Template.

Refactored <getContainerPorts> and moved port merge with error handling to it.

Removed a redundant <limit> put into the HashMap.

Merging Pod Template volume configs with Heron defaults. Heron values take precedence.

Allow user values for CPU and MEMORY limits to override those provided by Heron.

Some Schedulers, such as K8s, throw exceptions instead of returning false when <submit> fails. This leaves the Topology Manager with dangling references. An addition RPC call to the Scheduler is required to completely clear the state.

General cleanup in tests and class.

@nwangtw

Code review from @nwangtw. <KubernetesContext.getPodTemplateConfigMapDisabled> switched to <equalsIgnoreCase>.

@nwangtw

Code review from @nwangtw. <LaunchRunner> error message assembly improved. <LaunchRunner> added <FINE> level logging for failure to clear failed topology launch from Scheduler.

@nwangtw

Code review from @nwangtw, @nicknezis. Updated command to disable Pod Templates to <false> by default.

Logic for merging environment variables extracted to a method for testing.

Wired in <configureContainerEnvVars> and removed old code.

Logic to configure container's resources extracted to method to facilitate testing.

Removed old logic and wired <configureContainerResources> into <configureExecutorContainer>.

Added a test for debugging ports.

Exposed for testing.

Testing on a <hostPath> volume but will generalise across others.

Exposed for testing.

Tested by setting a Volume Mount in the Config and then a custom Volume Mount in the container.

Cleaned up tests.

Testing for when no Volume Mounts should be set.

Testing for when no Volumes should be set.

Extracted logic to generate executor environment variables, ports, and debugging ports. This is to resolve production-testing code inconsistencies which may arise.

General cleanup and simplification of test suite.

Added nested utility class to improve code maintainability. <mergeListDedupe> will merge two input lists by keeping all values in one and deduplicating the second list.

Full battery of tests null lists, merged lists, and thrown errors.

Switched to using <mergeListsDedupe> to improve code maintainability. Effects: <addVolumesIfPresent> <configureContainerEnvVars> <configureContainerPorts> <mountVolumeIfPresent>

…tter methods.

Merging Pod Specification Tolerations and deduplicating on the <V1Tolerations::key>.

Test for a null, empty, and merging of Toleration lists.

Wired in <configureTolerations>

Added check for multiple executor container specs in Pod Template. Will throw error if detected.

Updated tolerations to remove deprecated taints.

<getConfigMap> retrieving a single named ConfigMap in a specific namespace. <loadPodFromTemplate> logic updated to handle a single ConfigMap.

Fixed and cleaned up tests after switching to <readNamespacedConfigMap>.

Error message cleanup.

Test description cleanup.

Javadoc cleanup.

Test description cleanup.

<configureContainerResources> Heron values take precedence for limits.

… plugin. (apache#3712)

* pointing older releases to be retrieved from archive.a.o * typo

1. Install heron in the linux system to the /usr/lib/heron directory 2. Create symbolic links of several executable files in heron/bin to the /usr/bin directory 3. Try to call the heron* program separately, and the heron-downloader program fails to load the jar package correctly, and other programs are normal

Co-authored-by: Huijun Wu <huijunw@twitter.com>

Added enum for PVC option on CLI.

Creates a Map of volumes keyed on their names. Each volume name is then associated with a Map of options and their associated values.

Validate complete returned data structure.

<PersistentVolumeClaimOptions> switched to generic accessModes and added volumeMode.

Switched to a nested LinkedList because the Map is overkill and adds unneeded overhead.

<getPersistentVolumeClaims> basic logic to assemble a PVC with: volumeName (from CLI parameter) claimName (field) storageClassName (field) sizeLimit (field) accessModes (field) volumeMode (field)

Tests to generate a <PersistentVolumeClaimList> with single and multi access modes.

Refactored <configurePersistentVolumeClaims> to <createPersistentVolumeClaims> and returning a List now.

Wired in logic to create PVCs in <submit>.

Switched back to using a <Map<String, Map<KubernetesConstants.PersistentVolumeClaimOptions, String>>>. <EntrySet>, <KeySet>, and <values> methods return a view over the underlying data so there is no additional overhead. This change will facilitate the O(1) lookup of the <path> and <subPath> when creating the <Volume> and <VolumeMounts>. Methods effected: KubernetesContext.getPersistentVolumeClaims V1Controller.createPersistentVolumeClaims

<createPersistentVolumeClaimVolumes> will generate all Volumes from CLI PVC input.

Testing for empty and populated list of Volumes.

<createPersistentVolumeClaimVolumes> refactored to <createPersistentVolumeClaimVolumesAndMounts> and generates both the Volumes and Volume Mounts. They are circular prerequisites and it makes sense to combine them.

Added tests for Volume Mounts.

<configurePodWithPersistentVolumeClaims> will overwrite collisions in the Pod Spec and Executor. Will be employed before configuring the Executor or the Pod Spec.

Wired in <configurePodWithPersistentVolumeClaims> into <configurePodSpec>.

No Persistent Volume Claim test. Non-clashing Persistent Volume Claim test. Clashing Persistent Volume Claim test.

…umeClaim

<getPersistentVolumeClaimDisabled> will check for -D heron.kubernetes.persistent.volume.claims.cli.disabled=true

Added check for disabled PVC from the CLI into <submit>.

…umeClaim

Added entries to deployment scripts for: Minikube Kubernetes Helm Chart Added RBAC to create Persistent Volume Claims in Helm Chart.

…umeClaim

Fixed bug in <createPersistentVolumeClaims> where valid but unneeded fields were causing an error.

Style lint fix.

…umeClaim

Added support to generate a Dynamically backed PVC to <createPersistentVolumeClaims>. When the Claim Name is set to "OnDemand" a dynamic claim name will be generated and a <onDemand> key added to the options. The <onDemand> key will later be used to check if a PVC needs to be generated.

<createPersistentVolumeClaims>: added match labels and skipping of PVC creation for not "OnDemand" claims.

Added Javadocs for new functions.

<removeDynamicBackedPersistentVolumeClaims> Routine to connect to the K8s cluster and locate and remove all associated dynamically backed Persistent Volume Claims.

Wired <removeDynamicBackedPersistentVolumeClaims> into <killTopology> to remove all dynamically backed PVCs which were added by the topology.

@nwangtw

Reviewer(s): @nwangtw Removed nullptr check in <configurePodSpec>. Updated error message.

…umeClaim

Helm Chart permissions updated to support cleanup of dynamically backed Persistent Volume Claims.

Refactored <getPersistentVolumeClaimMatchLabels> to a more appropriate location.

Updates for dynamically and statically backed Persistent Volume Claims.

Added log statements in <removeDynamicBackedPersistentVolumeClaims>.

…umeClaim

Fixed bug in <generatePersistentVolumeClaimName> generating invalid names.

Updated <submit> with a more detailed error message.

Switched to <deleteCollectionNamespacedPersistentVolumeClaim> in <removeDynamicBackedPersistentVolumeClaims>.

Cleaned up matchLabels for Persistent Volume Claims.

…umeClaim

Added Labels to metadata in <createPersistentVolumeClaimVolumesAndMounts>.

…umeClaim

Streamlined <testGetPersistentVolumeClaimsOnDemand>.

Checking for failure conditions in <testGetPersistentVolumeClaimsErrors>.

Added <TestTuple> to simplify testing.

Simplified testing by switching to <TestTuple>.

Refactored <TestTuple> to make it more intuitive.

Cleaned up tests.

Updated javadoc and <removeDynamicBackedPersistentVolumeClaims> for labels.

Test suite cleanup:<testCreatePersistentVolumeClaimVolumesAndMounts> <testConfigurePodWithPersistentVolumeClaims>

…umeClaim

Added pattern matching for lowercase RFC-1123.

Adding lowercase RFC-1123 validation in <getPersistentVolumeClaims> for: - Volume Name - claimName - storageClassName Comment has been added to explain the evaluation of validation of: - claimName - storageClassName

Added tests to validate parameters to <getPersistentVolumeClaims>: - Volume Name - claimName - storageClassName

…umeClaim

CheckStyle formatting fix.

Made comment about validation more concise in <getPersistentVolumeClaims>. <notClaimNameOnDemand> switched to <claimNameNotOnDemand> to make it more intuitive.

…umeClaim

<VolumeClaimTemplateConfigKeys> Refactored and removed `onDemand` and `claimName`.

<getVolumeClaimTemplates> refactored and updated to handle changes to <VolumeClaimTemplateConfigKeys>.

Preliminary updates to handle change to <VolumeClaimTemplateConfigKeys>.

Updates for changes to <VolumeClaimTemplateConfigKeys>: <createPersistentVolumeClaimVolumeMounts> <configurePodWithPersistentVolumeClaims> <removePersistentVolumeClaims> <getPersistentVolumeClaimLabels>

Updated <configurePodWithPersistentVolumeClaims> for changes.

Updated for changes to support Volume Claim Templates: <createPersistentVolumeClaims>

Refactored to more meaningful name <configurePodWithPersistentVolumeClaimMounts>

Wired in <createPersistentVolumeClaims> to add PVC's to the StatefulSet Spec.

…umeClaim

…sistentVolumeClaim

Added back "claimName" to the <VolumeClaimTemplateConfigKeys>.

Added checks for `claimName`.

Added check for on-demand `claimName`s so that routine will only generate on-demand claims. I have switched over to performing a string comparison to reduce code complexity. Order of checks: Same object Same length Compare characters Given the above checks and the fixed length of `OnDemand` this should be O(1).

Updated tests for <createPersistentVolumeClaims>.

Added conditional Volume creation to <createPersistentVolumeClaimVolumeAndMounts>.

Updated to handle <Volumes>.

Updated <configurePodWithPersistentVolumeClaimVolumesAndMounts> to handle both <Volumes> and <VolumeMounts>.

<configurePodWithPersistentVolumeClaimVolumesAndMounts> adds both Volumes and Mounts.

…umeClaim

Weird indentation error indicating line is indented to column 12 instead of 13 when it is at 13. Added single space to line 902.

…umeClaim

Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>

Commits on Oct 21, 2021

[Tests] <V1Controller> general cleanup.

surahman committed Oct 21, 2021

Configuration menu

View commit details

Copy full SHA for ef2028c

Browse repository at this point

Copy the full SHA

ef2028c View commit details

Browse the repository at this point in the history

[Heron-3723] Add support for Persistent Volumes for stateful storage #3725

[Heron-3723] Add support for Persistent Volumes for stateful storage #3725

Commits on Sep 9, 2021

Commits on Sep 10, 2021

Commits on Sep 13, 2021

Commits on Sep 15, 2021

Commits on Sep 18, 2021

Commits on Sep 19, 2021

Commits on Sep 20, 2021

Commits on Sep 21, 2021

Commits on Sep 22, 2021

Commits on Sep 23, 2021

Commits on Sep 25, 2021

Commits on Sep 26, 2021

Commits on Sep 27, 2021

Commits on Sep 29, 2021

Commits on Sep 30, 2021

Commits on Oct 1, 2021

Commits on Oct 2, 2021

Commits on Oct 4, 2021

Commits on Oct 5, 2021

Commits on Oct 6, 2021

Commits on Oct 8, 2021

Commits on Oct 9, 2021

Commits on Oct 10, 2021

Commits on Oct 11, 2021

Commits on Oct 12, 2021

Commits on Oct 13, 2021

Commits on Oct 14, 2021

Commits on Oct 15, 2021

Commits on Oct 16, 2021

Commits on Oct 17, 2021

Commits on Oct 18, 2021

Commits on Oct 19, 2021

Commits on Oct 21, 2021

Commits on Oct 22, 2021

Commits on Oct 25, 2021

Commits on Oct 26, 2021

Commits on Oct 27, 2021

Commits on Oct 29, 2021

Commits on Oct 30, 2021

Commits on Oct 31, 2021

Commits on Nov 1, 2021

Commits on Nov 2, 2021

Commits on Nov 3, 2021

Commits on Nov 4, 2021

Commits on Nov 6, 2021

Commits on Nov 7, 2021

Commits on Nov 8, 2021

Commits on Nov 9, 2021

Commits on Nov 10, 2021

Commits on Nov 11, 2021

Commits on Nov 12, 2021

Commits on Nov 13, 2021

Commits on Nov 15, 2021

Commits on Nov 16, 2021

Commits on Nov 20, 2021

Commits on Nov 22, 2021

Commits on Nov 28, 2021