New syntax (#590)

* Fix errors

* kinda works

* scripts

* Fix some things

* Fix loading projects

* fix value access

* Fix source name

* Fix source name handling

* Remove git protocol enum

* Fix source names

* more stuff

* Separate source name from source

* Add back name property

* Fix recursive/submodules parameter name

* Add errors

* Fix protocol and yaml dump

* Got a working test stub

* Fix yaml output

* More test

* Add link step

* Add herd tags test

* Add herd commits test

* Add tags and project configurations

* Combine herd tests and add untracked file test

* Better clean check

* Add more common test utils

* Use common module prefix

* FIx upstream remote and loading

* Add test fixture

* Fix upstream herd

* Set default protocol for upstream

* Fix project urls with upstreams

* Relax rules for project list to allow strings

* Add github action workflow for pytest

* Add diff test

* Fix missing group names

* Fix tests in older python versions

* Fix some upstream logic

* Refactor tests layout

* Add variable length step argument

* Fix fork tests

* Fix protocol override

* Add back resolved yaml

* Add ability to get sha

* Convert branch and diff tests

* Convert status tests

* Save command results between steps

* Convert more tests

* Update scenarios

* Change defaults syntax

* Update protocol

* Schema and docs fixes

* Fix schema

* Fix test files

* Update syntax reference

* Convert forall tests

* Convert some subdirectory tests

* Add parallel forall test

* Convert forall upstream tests

* Convert checkout tests

* Convert link tests

* Convert save tests

* Use https version

* Convert prune tests

* Fix checkout tests and test output

* Convert reset tests

* Upstream protocol fixes

* Convert remaining reset tests

* Convert start tests

* Rename step variable

* Convert stash tests

* Convert parallel reset tests

* Convert yaml tests

* Refactor fixtures

* Reorganize test files

* More reorganizing

* Split up given steps

* Consolidate when steps

* Refactor then steps

* Refactor test utils

* Convert repo tests

* Convert herd submodule tests

* Update docs

* Convert misc defaults tests

* Convert more misc tests

* Remove workflow

* Add more PyCharm configurations

* Renaming

* Refactor given steps

* Rename

* Convert init tests

* Refactor step names and more

* Rename TestInfo to ScenarioInfo

* Convert more init tests

* moar

* even moar

* Remove old init scripts

* Fix github actions workflow for pytest-bdd serial

* Convert groups tests

* Convert offline tests

* more

* Update naming and configurations

* even moar

* more

* Comment out repo offline tests

* doesn't work

* offline tests work

* Remove serial marker

* Disable offline tests

* clean up

* Fix case

* Fix start tracking test

* More fine grained network connectivity

* Fix herd rebase bug with branch/tag sharing a name

* Remove write herd script

* Add prune write tests, fixes, remove redundant table columns

* Convert prune write tests

* Remove success marker

* Better typing

* Convert subdirectory tests

* Convert start write tests

* Move old scripts to directory and remove old test workflows

* Rename workflow

* Move files to root

* Remove src paths and clowder_test

* Remove noinspection

* Fix clean script calls

* Update circle ci config

* Fix scripts

* Rename file_name -> filename

* Clean up tests

* Coverage works

* Update ci scripts

* try string

* Fix CI

* Print CI info

* Fix paths

* Save artifacts

* Try fixing paths

* Don't rewrite paths

* Try more path stuff

* Convert more herd tests

* moar

* some moar

* Convert remaining herd test

* Fix tests

* Remove escape sequences from local test output

* Consolidate test script

* Rename reports

* store more test results

* Enable junit xml

* Add name argument to test script

* Run more tests on circle

* Use ssh for write tests

* Add codeclimate.yml

* Remove example converters

* Refactor some given steps

* Save config file in .clowder directory

* Fix tests

* Converrt some clean tests

* More clean tests

* Fix lint issues

* Add clean test for abort rebase

* Update code climate yml

* Loosen code climate requirements

* Disable code climate plugins

* code climate changes

* Convert lfs tests

* lfs uninstall

* Revert "lfs uninstall"

This reverts commit c6e787e.

* Disable lfs

* Force delete remote branch

* Try uninstalling more lfs

* Debug CI

* Install lfs on circle

* Fix delete remote branch

* More lfs

* uninstall global lfs

* Uninstall global lfs in circle

* Use sudo

* unset all

* system git config

* use sudo

* system

* Add debug output

* unset all lfs

* Revert "unset all lfs"

This reverts commit 4eb9fd7.

* install lfs locally

* Fix local lfs install

* Re-enable circle tests

* Debug

* try to fix test

* debug

* Uninstall lfs config in github actions workflow

* Delete remote pytest branches in fixture

* Re-enable circle tests

* Only delete remote branches with ssh

* Revert "Re-enable circle tests"

This reverts commit 9a70f49.

* Revert "Revert "Re-enable circle tests""

This reverts commit c6b0e1f.

* Convert config tests

* Add comments

* debug

* Update test branch names

* fetch more often

* Update doc comments

* Update some errors

* Convert git config tests

* Convert herd branch tests

* Create yaml syntax test files

* Configured most of project test files

* Fix test

* Fix yaml tests

* Convert herd tag tests

* More test data updates

* Fix test

* Add parallel command TODOs

* Update test data

* Update group project test data

* Update group projects test data

* Organize directories

* Update repos used in test data

* Remove sources and protocol from group options

* Start on first validation test

* Working branch validation test

* More modular test

* Add remaining yaml validation tests

* wip

* More yaml validation tests implemented

* Fix failing tests

* Add tests for depth

* debug circle ci

* Revert "debug circle ci"

This reverts commit 3938366.

* Separate submodules from recursive

* Remove need for no-versions branch

* Remove need for duplicate-version branch

* Remove need for yaml-validation branch

* Add old invalid yaml versions

* Remove need for repo-test and extension branches

* Remove unused given steps

* Add ssh key to circle

* try anothe ssh key

* Add key to known_hosts

* Restore circle tests

* Remove yaml -f option

* Fix init tests

* Fix link test

* Run circle tests in parallel

* Simplify gh action pytest workflow

* Add postfix to report files

* Fix yaml output tests

.circleci/config.yml

@@ -3,20 +3,20 @@
 # Check https://circleci.com/docs/2.0/language-python/ for more details
 #
 version: 2.1
+orbs:
+  codecov: codecov/codecov@1.0.2
 jobs:
   build:
     docker:
       - image: circleci/python:3.7.7
 
-    working_directory: ~/clowder
-
     steps:
       - checkout
 
       # Download and cache dependencies
       - restore_cache:
           keys:
-          - v1-dependencies-{{ checksum "src/requirements.txt" }}
+          - v1-dependencies-{{ checksum "requirements.txt" }}
           # fallback to using the latest cache if no exact match is found
           - v1-dependencies-
 
@@ -25,19 +25,17 @@ jobs:
           command: |
             python3 -m venv venv
             . venv/bin/activate
-            sudo pip3 install -r src/requirements.txt
+            sudo pip3 install -r requirements.txt
+            sudo apt-get install git-lfs
 
       - run:
           name: run setup scripts
-          command: |
-            script/ci_before
-            script/update
-            script/test
+          command: script/update
 
       - save_cache:
           paths:
             - ./venv
-          key: v1-dependencies-{{ checksum "src/requirements.txt" }}
+          key: v1-dependencies-{{ checksum "requirements.txt" }}
 
       - run:
           name: setup git config
@@ -46,16 +44,55 @@ jobs:
             git config --global user.name "CircleCI"
             git config --global push.default simple
             git config --global --unset url.ssh://git@github.com.insteadOf
+            sudo git config --system --unset-all filter.lfs.clean
+            sudo git config --system --unset-all filter.lfs.smudge
+            sudo git config --system --unset-all filter.lfs.process
+            sudo git config --system --unset-all filter.lfs.required
 
-      - run:
-          name: run write tests
-          command: clowder-test -c write
+      - add_ssh_keys:
+          fingerprints:
+            - "e9:7d:20:a0:51:b2:9c:22:f6:b9:5b:55:47:c4:7c:20"
 
       - run:
-          name: run misc tests
-          command: clowder-test -c misc all
+          name: Add ssh key to known_hosts
+          command:  ssh-keyscan polka-dot-cat.git.beanstalkapp.com >> ~/.ssh/known_hosts
 
       - run:
-          name: upload code coverage results
+          name: Print environment info
           command: |
-            script/ci_after
+            pwd
+            ls -al
+
+      - run:
+          name: Code climate setup
+          command: script/ci_before
+
+      - run:
+          name: run pytest functional tests - parallel
+          command: script/test "-n auto -m 'not write and not offline'"
+
+      - run:
+          name: run pytest functional tests - write
+          command: script/test "-m 'write and not offline'"
+
+# TODO: Enable offline tests
+# https://discuss.circleci.com/t/running-tests-without-internet-access/1427
+      # - run:
+      #     name: run pytest functional tests - offline
+      #     command: script/test "-m 'offline'"
+
+      - run:
+          name: Code climate reporting
+          command: script/ci_after
+
+      - store_test_results:
+          path: build/reports
+
+      - codecov/upload:
+          file: build/coverage/.coverage
+
+      - store_artifacts:
+          path: build/coverage
+
+      - store_artifacts:
+          path: build/reports

.codeclimate.yml

@@ -0,0 +1,59 @@
+version: "2"
+
+# engines:
+#   pep8:
+#     enabled: true
+#   shellcheck:
+#     enabled: true
+#   sonar-python:
+#     enabled: true
+  # fixme:
+  #   enabled: true
+  # TODO: https://docs.codeclimate.com/docs/radon
+  # radon:
+  #   enabled: true
+  # TODO: https://docs.codeclimate.com/docs/bandit
+  # bandit:
+  #   enabled: true
+
+ratings:
+  paths:
+  - "**.py"
+  - "**.sh"
+
+
+checks:
+  argument-count:
+    config:
+      threshold: 8
+  complex-logic:
+    config:
+      threshold: 40
+  file-lines:
+    config:
+      threshold: 1000
+  method-complexity:
+    config:
+      threshold: 30
+  method-count:
+    config:
+      threshold: 80
+  method-lines:
+    config:
+      threshold: 50
+  nested-control-flow:
+    config:
+      threshold: 4
+  return-statements:
+    config:
+      threshold: 8
+  # similar-code:
+  #   config:
+  #     threshold: 20
+  # identical-code:
+  #   config:
+  #     threshold: 20
+
+# exclude_patterns:
+#   - "tests/"
+#   - "script/"

.coveragerc

@@ -2,14 +2,20 @@
 branch = True
 concurrency = multiprocessing
 parallel = True
-# data_file = coverage/.coverage
+dynamic_context = test_function
+data_file = build/coverage/.coverage
 omit =
-    # omit everything in /usr
-    /usr/*
+    tests/*
+    clowder/sitecustomize.py
+source = clowder/*
+# debug = config
 
 [paths]
 source =
-    src/clowder
+    clowder/
+    /Users/runner/work/clowder/clowder/clowder/
+    /home/circleci/project/clowder/
+    /Users/jrgoodle/projects/clowder-projects/clowder/clowder/
 
 [report]
 # Regexes for lines to exclude from consideration