Python: Extract hidden files/dirs by default

Changes the default behaviour of the Python extractor so that hidden
files (and files inside hidden directories) are extracted by default.

Also adds an extractor option, `skip_hidden_files`, which can be set to
`true` in order to revert to the old behaviour.

Finally, I made the logic surrounding what is logged in various cases a
bit more obvious.

Technically this changes the behaviour of the extractor (in that hidden
excluded files will now be logged as `(excluded)`, but I think this
makes more sense anyway.

Author: tausbn

python/codeql-extractor.yml

@@ -44,3 +44,10 @@ options:
       Use this setting with caution, the Python extractor requires Python 3 to run.
     type: string
     pattern: "^(py|python|python3)$"
+  skip_hidden_files:
+    title: Controls whether hidden files and directories are skipped during extraction.
+    description: >
+      By default, CodeQL will extract all Python files, including hidden files and ones located in hidden directories. By setting this option to true, these hidden files and directories will be skipped instead.
+      Accepted values are true and false.
+    type: string
+    pattern: "^(true|false)$"

python/extractor/semmle/traverser.py

@@ -83,11 +83,10 @@ def _treewalk(self, path):
                 self.logger.debug("Ignoring %s (symlink)", fullpath)
                 continue
             if isdir(fullpath):
-                if fullpath in self.exclude_paths or is_hidden(fullpath):
-                    if is_hidden(fullpath):
-                        self.logger.debug("Ignoring %s (hidden)", fullpath)
-                    else:
-                        self.logger.debug("Ignoring %s (excluded)", fullpath)
+                if fullpath in self.exclude_paths:
+                    self.logger.debug("Ignoring %s (excluded)", fullpath)
+                elif is_hidden(fullpath):
+                    self.logger.debug("Ignoring %s (hidden)", fullpath)
                 else:
                     empty = True
                     for item in self._treewalk(fullpath):
@@ -101,7 +100,12 @@ def _treewalk(self, path):
                 self.logger.debug("Ignoring %s (filter)", fullpath)
 
 
-if os.name== 'nt':
+if os.environ.get("CODEQL_EXTRACTOR_PYTHON_SKIP_HIDDEN_FILES", "false") == "false":
+
+    def is_hidden(path):
+        return False
+
+elif os.name== 'nt':
     import ctypes
 
     def is_hidden(path):