Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release v1.3.0 #11

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

release v1.3.0 #11

wants to merge 14 commits into from

Conversation

marcellodesales
Copy link
Member

Features

  • Python3 validator implementation
  • package.sh: users can now package an existing version or a version from the source-code in 1 single command

marcellodesales and others added 14 commits November 26, 2019 01:43
@marcellodesales
Merge pull request #9 from intuit/master
3971f1b 
Rebase
@marcellodesales
Issue #8: Python3: converting using 2to3 validator
cf0dff6 
Just ran the following command:

2to3 -n -W --add-suffix=3 validate_config_files.py

A new file was created validate_config_files.py3 and I just
replaced the existing with this version.

https://docs.python.org/2/library/2to3.html
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Dockerfile: Using pip instead of pip2: use python3
062b2ce 
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Tests: using 2to3 migration of test cases
85fd363 
Just minor changes

$ 2to3 --output-dir=tests3 -W -n tests
WARNING: --write-unchanged-files/-W implies -w.
lib2to3.main: Output in 'tests3' will mirror the input directory 'tests' layout.
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored tests/context.py
--- tests/context.py	(original)
+++ tests/context.py	(refactored)
@@ -43,12 +43,12 @@
   message = str(isValid) if isValid else str(isValid) + " ERROR: " + str(validationObject)

   # The V of successful in green
-  v = ShellColor.OKGREEN + str(u'\u2714'.encode('UTF-8'))
+  v = ShellColor.OKGREEN + str('\u2714'.encode('UTF-8'))
   # The X of failure in red
-  x = ShellColor.FAIL + str(u'\u2718'.encode('UTF-8'))
+  x = ShellColor.FAIL + str('\u2718'.encode('UTF-8'))
   shellStatus = v if isValid else x

-  print shellStatus + " is " + getRelativeFixturePath(filePath) + " valid? " + message + ShellColor.ENDC
+  print(shellStatus + " is " + getRelativeFixturePath(filePath) + " valid? " + message + ShellColor.ENDC)

 class ValidationAssertions:
   """Validation Assertions for the test cases to validate on the values."""
RefactoringTool: Writing converted tests/context.py to tests3/context.py.
RefactoringTool: Refactored tests/test_all_valid_config_validation.py
--- tests/test_all_valid_config_validation.py	(original)
+++ tests/test_all_valid_config_validation.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_all_properties_are_valid(self):
-    print ShellColor.WARNING + "All config files are valid" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "All config files are valid" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       printFileValidationStatus(filePath, validationObject)

       # Verify if the directory is in the file path
RefactoringTool: Writing converted tests/test_all_valid_config_validation.py to tests3/test_all_valid_config_validation.py.
RefactoringTool: Refactored tests/test_invalid_matrix_json_validation.py
--- tests/test_invalid_matrix_json_validation.py	(original)
+++ tests/test_invalid_matrix_json_validation.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_all_matrix_json_files_are_invalid(self):
-    print ShellColor.WARNING + "The android matrix file is invalid" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "The android matrix file is invalid" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       isValid = isConfigValid(validationObject)
       printFileValidationStatus(filePath, validationObject)

RefactoringTool: Writing converted tests/test_invalid_matrix_json_validation.py to tests3/test_invalid_matrix_json_validation.py.
RefactoringTool: Refactored tests/test_invalid_properties_validation.py
--- tests/test_invalid_properties_validation.py	(original)
+++ tests/test_invalid_properties_validation.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_some_yaml_yml_files_are_invalid(self):
-    print ShellColor.WARNING + "Properties files are invalid without associated values" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "Properties files are invalid without associated values" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       isValid = isConfigValid(validationObject)
       printFileValidationStatus(filePath, validationObject)

RefactoringTool: Writing converted tests/test_invalid_properties_validation.py to tests3/test_invalid_properties_validation.py.
RefactoringTool: Refactored tests/test_invalid_yaml_yml_duplicate_keys.py
--- tests/test_invalid_yaml_yml_duplicate_keys.py	(original)
+++ tests/test_invalid_yaml_yml_duplicate_keys.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_some_yaml_yml_files_are_invalid(self):
-    print ShellColor.WARNING + "Some Yaml Single documents are invalid" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "Some Yaml Single documents are invalid" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       isValid = isConfigValid(validationObject)
       printFileValidationStatus(filePath, validationObject)

RefactoringTool: Writing converted tests/test_invalid_yaml_yml_duplicate_keys.py to tests3/test_invalid_yaml_yml_duplicate_keys.py.
RefactoringTool: Refactored tests/test_invalid_yaml_yml_multi_document_validation.py
--- tests/test_invalid_yaml_yml_multi_document_validation.py	(original)
+++ tests/test_invalid_yaml_yml_multi_document_validation.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_some_yaml_yml_files_are_invalid(self):
-    print ShellColor.WARNING + "Some Yaml Multi documents are invalid" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "Some Yaml Multi documents are invalid" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       isValid = isConfigValid(validationObject)
       printFileValidationStatus(filePath, validationObject)

RefactoringTool: Writing converted tests/test_invalid_yaml_yml_multi_document_validation.py to tests3/test_invalid_yaml_yml_multi_document_validation.py.
RefactoringTool: Refactored tests/test_invalid_yaml_yml_single_document_validation.py
--- tests/test_invalid_yaml_yml_single_document_validation.py	(original)
+++ tests/test_invalid_yaml_yml_single_document_validation.py	(refactored)
@@ -26,8 +26,8 @@
     self.assertTrue(len(self.validationIndex) > 0)

   def test_some_yaml_yml_files_are_invalid(self):
-    print ShellColor.WARNING + "Some Yaml Single documents are invalid" + ShellColor.ENDC
-    for filePath, validationObject in self.validationIndex.iteritems():
+    print(ShellColor.WARNING + "Some Yaml Single documents are invalid" + ShellColor.ENDC)
+    for filePath, validationObject in self.validationIndex.items():
       isValid = isConfigValid(validationObject)
       printFileValidationStatus(filePath, validationObject)

RefactoringTool: Writing converted tests/test_invalid_yaml_yml_single_document_validation.py to tests3/test_invalid_yaml_yml_single_document_validation.py.
RefactoringTool: Files that were modified:
RefactoringTool: tests/context.py
RefactoringTool: tests/test_all_valid_config_validation.py
RefactoringTool: tests/test_invalid_matrix_json_validation.py
RefactoringTool: tests/test_invalid_properties_validation.py
RefactoringTool: tests/test_invalid_yaml_yml_duplicate_keys.py
RefactoringTool: tests/test_invalid_yaml_yml_multi_document_validation.py
RefactoringTool: tests/test_invalid_yaml_yml_single_document_validation.py
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Tests: python 3 must add the UTF-8 param to str for decoding
a6a40ca 
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Validate: Python3: Closing file resource: reading yaml duri…
50ea725 
…ng lint

Getting the following error:

/build/validate_config_files.py:181: ResourceWarning: unclosed file <_io.TextIOWrapper name='/build/tests/fixtures/all-valid-config/publisher-onboard_preprod.yml' mode='r' encoding='UTF-8'>
  lintError = list(linter.run(open(filePath), yamlLintConfig));

Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Dockerfile: Python3 hide deprecation warnings: glob2: syste…
1be4255 
…m-wide

https://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python/55947748#55947748

test_all_properties_are_valid (test_all_valid_config_validation.AllSuccessfulTests) ... /usr/lib/python3.6/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\Z(?ms)'
  return re.compile(res, flags).match
/usr/lib/python3.6/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\.json\\Z(?ms)'
  return re.compile(res, flags).match
/usr/lib/python3.6/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\.yaml\\Z(?ms)'
  return re.compile(res, flags).match
/usr/lib/python3.6/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\.yml\\Z(?ms)'
  return re.compile(res, flags).match
/usr/lib/python3.6/site-packages/glob2/fnmatch.py:80: DeprecationWarning: Flags not at the start of the expression '(.*)\\.properties\\Z(?' (truncated)
  return re.compile(res, flags).match
ok

Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Validate: initial setup failing: object str byte
bd64de8 
Fixed by using the decode UTF-8 method
https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write/21689447#21689447

remote: #####################################################
remote: #### Intuit Spring Cloud Config Validator 1.2.0 #####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=5f2acca1293550dc4ecc7aab26e77c52d74f15aa ref=refs/heads/master
remote: Traceback (most recent call last):
remote:   File "hooks/pre-receive", line 354, in <module>
remote:     (executationDirPath, validationIndex) = ShellExecution.run()
remote:   File "hooks/pre-receive", line 326, in run
remote:     currentDirPath = dirPath if dirPath else ExecutionContext.getCurrentDirPath()
remote:   File "hooks/pre-receive", line 83, in getCurrentDirPath
remote:     currentDirPath = Validator.processPreReceivehookFilesInGithub(base, commit)
remote:   File "hooks/pre-receive", line 243, in processPreReceivehookFilesInGithub
remote:     content = GitRepo.openCommitFileContent(fileName, head)
remote:   File "hooks/pre-receive", line 129, in openCommitFileContent
remote:     (results, code) = GitRepo.git(('git', 'show', commit + ":" + fileName))
remote: TypeError: must be str, not bytes
To 192.168.86.249:test.git

Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Validate: python3: bytes to string on github stream
b8643fd 
Fixed with the decode method described at
https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write/21689447#21689447

* Broken issue:

GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git push test new-branch -f
Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
Enumerating objects: 37, done.
Counting objects: 100% (37/37), done.
Delta compression using up to 8 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (35/35), 1.93 KiB | 659.00 KiB/s, done.
Total 35 (delta 8), reused 0 (delta 0)
remote: #####################################################
remote: #### Intuit Spring Cloud Config Validator 1.3.0 #####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=a34a90b2bfcd2018b746e8ee78e51bf393e6dbc1 ref=refs/heads/new-branch
remote: Traceback (most recent call last):
remote:   File "hooks/pre-receive", line 354, in <module>
remote:     (executationDirPath, validationIndex) = ShellExecution.run()
remote:   File "hooks/pre-receive", line 326, in run
remote:     currentDirPath = dirPath if dirPath else ExecutionContext.getCurrentDirPath()
remote:   File "hooks/pre-receive", line 83, in getCurrentDirPath
remote:     currentDirPath = Validator.processPreReceivehookFilesInGithub(base, commit)
remote:   File "hooks/pre-receive", line 263, in processPreReceivehookFilesInGithub
remote:     filePath = Validator.saveFileContent(fileName, content, contextDir)
remote:   File "hooks/pre-receive", line 202, in saveFileContent
remote:     text_file.write(content)
remote: TypeError: write() argument must be str, not bytes

Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: python3: Env: update version to 1.3.0
c261e3e 
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: python3: README: add link and asciirec to package.sh
324bbc4 
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: Details about package.sh for Hook packaging
b14054a 
For packaging for Github Enterprise, users can now
package from the current source or package from the
given docker image version specified in the .env file.

Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Issue #8: README: details on the pull vs pacakge.sh:
2eaa2df 
Signed-off-by: Marcello de Sales <Marcello_deSales@intuit.com>
@marcellodesales
Merge pull request #10 from intuit/feature/migrate-python-3
854a9d6 
Migrate to python 3 + packaging to Github Enterprise
@marcellodesales marcellodesales added this to the v1.3.0 milestone Nov 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.3.0
Development

Successfully merging this pull request may close these issues.
1 participant
@marcellodesales