Feature: Added fetch_always functionality

docs/external_dependencies.md

@@ -32,10 +32,10 @@ $ kapitan compile --fetch
 This will download the dependencies and store them at their respective `output_path`.
 By default, kapitan does not overwrite existing items with the same name as that of the fetched dependencies.
 
-Use the `--force` flag to force fetch (update cache with freshly fetched dependencies) and overwrite any existing item sharing the same name in the `output_path`.
+Use the `--force-fetch` flag to force fetch (update cache with freshly fetched dependencies) and overwrite any existing item sharing the same name in the `output_path`.
 
 ```
-$ kapitan compile --fetch --force
+$ kapitan compile --force-fetch
 ```
 
 Use the `--cache` flag to cache the fetched items in the `.dependency_cache` directory in the root project directory.

docs/inventory.md

@@ -277,10 +277,10 @@ $ kapitan compile --fetch
 This will download the dependencies and store them at their respective `output_path`.
 By default, kapitan does not overwrite an existing item with the same name as that of the fetched inventory items.
 
-Use the `--force` flag to force fetch (update cache with freshly fetched items) and overwrite inventory items of the same name in the `output_path`.
+Use the `--force-fetch` flag to force fetch (update cache with freshly fetched items) and overwrite inventory items of the same name in the `output_path`.
 
 ```
-$ kapitan compile --fetch --force
+$ kapitan compile --fetch --force-fetch
 ```
 
 Use the `--cache` flag to cache the fetched items in the `.dependency_cache` directory in the root project directory.

docs/kap_proposals/kap_1_external_dependencies.md

@@ -27,6 +27,7 @@ Git type may also include `ref` and `subdir` parameters as illustrated below:
   source: <git_url>
   subdir: relative/path/in/repository
   ref: <commit_hash/branch/tag>
+  force_fetch: <bool>
 ```
 
 If the file already exists at `output_path`, the fetch will be skipped. For fresh fetch of the dependencies, users may add `--fetch` option as follows:
@@ -35,12 +36,12 @@ If the file already exists at `output_path`, the fetch will be skipped. For fres
 $ kapitan compile --fetch
 ```
 
-Users can also add the `fetch_always: true` option to the `kapitan.compile` in the inventory in order to force fresh fetch of the dependencies every time.
+Users can also add the `force_fetch: true` option to the `kapitan.dependencies` in the inventory in order to force fetch of the dependencies of the target every time.
 
 ## Implementation details
 
 ### Dependencies
 
 - [GitPython](https://github.com/gitpython-developers/GitPython) module (and git executable) for git type
 - requests module for http[s]
-- (optional) [tqdm](https://github.com/tqdm/tqdm) for reporting download progress
+- (optional) [tqdm](https://github.com/tqdm/tqdm) for reporting download progress

docs/kap_proposals/kap_7_remote_inventory.md

@@ -31,9 +31,9 @@ eg. `$ kapitan compile --fetch --cache`
 ## Force fetching
 While fetching, the output path will be recursively checked to see if it contains any file with the same name. If so, kapitan will skip fetching it.
 
-To overwrite the files with the newly downloaded inventory items, we can add the `--force` flag to the compile command, as shown below.
+To overwrite the files with the newly downloaded inventory items, we can add the `--force-fetch` flag to the compile command, as shown below.
 
-`$ kapitan compile --fetch --force`
+`$ kapitan compile --force-fetch`
 
 ## URL type
 The URL type can be either git or http(s). Depending on the URL type, the configuration file may have additional arguments.

docs/usage.md

@@ -57,7 +57,7 @@ optional arguments:
   --quiet               set quiet mode, only critical output
   --output-path PATH    set output path, default is "."
   --fetch               fetch remote inventories and/or external dependencies
-  --force               overwrite existing inventory and/or dependency item
+  --force-fetch         overwrite existing inventory and/or dependency item
   --validate            validate compile output against schemas as specified
                         in inventory
   --parallelism INT, -p INT

kapitan/cli.py

@@ -85,9 +85,9 @@ def trigger_compile(args):
         reveal=args.reveal,
         cache=args.cache,
         cache_paths=args.cache_paths,
-        fetch_inventories=args.fetch,
-        fetch_dependencies=args.fetch,
-        force_fetch=args.force,
+        fetch=args.fetch,
+        force_fetch=args.force_fetch,
+        force=args.force,  # deprecated
         validate=args.validate,
         schemas_path=args.schemas_path,
         jinja2_filters=args.jinja2_filters,
@@ -184,6 +184,12 @@ def build_parser():
         default=from_dot_kapitan("compile", "fetch", False),
     )
     compile_parser.add_argument(
+        "--force-fetch",
+        help="overwrite existing inventory and/or dependency item",
+        action="store_true",
+        default=from_dot_kapitan("compile", "force-fetch", False),
+    )
+    compile_parser.add_argument(  # deprecated
         "--force",
         help="overwrite existing inventory and/or dependency item",
         action="store_true",

kapitan/targets.py

@@ -80,7 +80,19 @@ def compile_targets(
 
     try:
         rendering_start = time.time()
-        if kwargs.get("fetch_inventories", False):
+
+        # check if --fetch or --force-fetch is enabled
+        force_fetch = kwargs.get("force_fetch", False)
+        fetch = kwargs.get("fetch", False) or force_fetch
+
+        # deprecated --force flag
+        if kwargs.get("force", False):
+            logger.info(
+                "DeprecationWarning: --force is deprecated. Use --force-fetch instead of --force --fetch"
+            )
+            force_fetch = True
+
+        if fetch:
             # skip classes that are not yet available
             target_objs = load_target_inventory(inventory_path, updated_targets, ignore_class_notfound=True)
         else:
@@ -93,15 +105,16 @@ def compile_targets(
         if not target_objs:
             raise CompileError("Error: no targets found")
 
-        if kwargs.get("fetch_inventories", False):
+        # fetch inventory
+        if fetch:
             # new_source checks for new sources in fetched inventory items
             new_sources = list(set(list_sources(target_objs)) - cached.inv_sources)
             while new_sources:
                 fetch_inventories(
                     inventory_path,
                     target_objs,
                     dep_cache_dir,
-                    kwargs.get("force_fetch", False),
+                    force_fetch,
                     pool,
                 )
                 cached.reset_inv()
@@ -113,10 +126,29 @@ def compile_targets(
             # reset inventory cache and load target objs to check for missing classes
             cached.reset_inv()
             target_objs = load_target_inventory(inventory_path, updated_targets, ignore_class_notfound=False)
-        if kwargs.get("fetch_dependencies", False):
-            fetch_dependencies(
-                output_path, target_objs, dep_cache_dir, kwargs.get("force_fetch", False), pool
-            )
+        # fetch dependencies
+        if fetch:
+            fetch_dependencies(output_path, target_objs, dep_cache_dir, force_fetch, pool)
+        # fetch targets which have force_fetch: true
+        elif not kwargs.get("force_fetch", False):
+            fetch_objs = []
+            # iterate through targets
+            for target in target_objs:
+                try:
+                    # get value of "force_fetch" property
+                    dependencies = target["dependencies"]
+                    # dependencies is still a list
+                    for entry in dependencies:
+                        force_fetch = entry["force_fetch"]
+                        if force_fetch:
+                            fetch_objs.append(target)
+                except KeyError:
+                    # targets may have no "dependencies" or "force_fetch" key
+                    continue
+            # fetch dependencies from targets with force_fetch set to true
+            if fetch_objs:
+                fetch_dependencies(output_path, fetch_objs, dep_cache_dir, True, pool)
+
         logger.info("Rendered inventory (%.2fs)", time.time() - rendering_start)
 
         worker = partial(
@@ -620,6 +652,7 @@ def valid_target_obj(target_obj, require_compile=True):
                         "ref": {"type": "string"},
                         "unpack": {"type": "boolean"},
                         "version": {"type": "string"},
+                        "force_fetch": {"type": "boolean"},
                     },
                     "required": ["type", "output_path", "source"],
                     "additionalProperties": False,
@@ -632,6 +665,7 @@ def valid_target_obj(target_obj, require_compile=True):
                                     "source": {"format": "uri"},
                                     "output_path": {},
                                     "unpack": {},
+                                    "force_fetch": {},
                                 },
                                 "additionalProperties": False,
                             },
@@ -646,6 +680,7 @@ def valid_target_obj(target_obj, require_compile=True):
                                     "unpack": {},
                                     "chart_name": {"type": "string"},
                                     "version": {"type": "string"},
+                                    "force_fetch": {},
                                 },
                                 "required": ["type", "output_path", "source", "chart_name"],
                                 "additionalProperties": False,

tests/test_remote_inventory.py

@@ -90,7 +90,7 @@ def test_unpack_http_inv(self):
         rmtree(output_dir)
 
     def test_compile_fetch(self):
-        """Run $ kapitan compile --fetch --output-path=some/dir/ --inventory-path=another/dir --targets remoteinv-example remoteinv-nginx zippedinv --force
+        """Run $ kapitan compile --force-fetch --output-path=some/dir/ --inventory-path=another/dir --targets remoteinv-example remoteinv-nginx zippedinv
         were some/dir/ & another/dir/ are directories chosen by the user
 
         Recursively force fetch inventories and compile
@@ -110,7 +110,7 @@ def test_compile_fetch(self):
         sys.argv = [
             "kapitan",
             "compile",
-            "--fetch",
+            "--force-fetch",
             "--output-path",
             temp_output,
             "--inventory-path",
@@ -119,7 +119,6 @@ def test_compile_fetch(self):
             "remoteinv-example",
             "remoteinv-nginx",
             "zippedinv",
-            "--force",
         ]
         main()
 
@@ -159,10 +158,10 @@ def test_compile_fetch_classes_that_doesnot_exist_yet(self):
         rmtree(temp_dir)
 
     def test_force_fetch(self):
-        """Test overwriting inventory items while using the --force flag
+        """Test overwriting inventory items while using the --force-fetch flag
 
         runs $ kapitan compile --fetch --cache --output-path=temp_output --inventory-pat=temp_inv --targets remoteinv-example
-        runs $ kapitan compile --fetch --cache --output-path=temp_output --inventory-pat=temp_inv --targets remoteinv-example --force
+        runs $ kapitan compile --force-fetch --cache --output-path=temp_output --inventory-pat=temp_inv --targets remoteinv-example
         """
 
         temp_output = tempfile.mkdtemp()
@@ -205,15 +204,14 @@ def test_force_fetch(self):
         sys.argv = [
             "kapitan",
             "compile",
-            "--fetch",
+            "--force-fetch",
             "--cache",
             "--output-path",
             temp_output,
             "--inventory-path",
             temp_inv,
             "--targets",
             "remoteinv-example",
-            "--force",
         ]
         main()
         # zippedinv.yml overwritten