Merge https://github.com/cqfn/aibolit into fix_multiple_while

README.md

@@ -1,12 +1,12 @@
 <img src="/logo.svg" height="92px"/>
 
 [![PyPi version](https://img.shields.io/pypi/v/aibolit.svg)](https://pypi.org/project/aibolit/)
-[![Build Status](https://travis-ci.org/yegor256/aibolit.svg)](https://travis-ci.org/yegor256/aibolit)
-[![Build status](https://ci.appveyor.com/api/projects/status/1k7q7eumnhia0e3a?svg=true)](https://ci.appveyor.com/project/yegor256/aibolit)
-[![Hits-of-Code](https://hitsofcode.com/github/yegor256/aibolit)](https://hitsofcode.com/view/github/yegor256/aibolit)
-[![Test Coverage](https://img.shields.io/codecov/c/github/yegor256/aibolit.svg)](https://codecov.io/github/yegor256/aibolit?branch=master)
-[![Maintainability](https://api.codeclimate.com/v1/badges/e90e80a143a9457ee3af/maintainability)](https://codeclimate.com/github/yegor256/aibolit/maintainability)
-[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/yegor256/aibolit/blob/master/LICENSE.txt)
+[![Build Status](https://travis-ci.org/cqfn/aibolit.svg)](https://travis-ci.org/cqfn/aibolit)
+[![Build status](https://ci.appveyor.com/api/projects/status/gr3eqxq5pr87kpwg?svg=true)](https://ci.appveyor.com/project/yegor256/aibolit)
+[![Hits-of-Code](https://hitsofcode.com/github/cqfn/aibolit)](https://hitsofcode.com/view/github/cqfn/aibolit)
+[![Test Coverage](https://img.shields.io/codecov/c/github/cqfn/aibolit.svg)](https://codecov.io/github/cqfn/aibolit?branch=master)
+[![Maintainability](https://api.codeclimate.com/v1/badges/fd7e32d8472b4d5e8ecb/maintainability)](https://codeclimate.com/github/cqfn/aibolit/maintainability)
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/cqfn/aibolit/blob/master/LICENSE.txt)
 
 First, you install it (you must have [Python 3.7.7](https://www.python.org/downloads/)
 and [Pip](https://pip.pypa.io/en/stable/installing/) installed):
@@ -15,8 +15,6 @@ and [Pip](https://pip.pypa.io/en/stable/installing/) installed):
 $ pip3 install aibolit
 ```
 
-### Recommend command
-
 To analyze your Java sources, located at `src/java` (for example), run:
 
 ```bash
@@ -34,11 +32,11 @@ Also, you can set a folder with Java files:
 $ aibolit recommend --folder src/java
 ```
 
-It will run recommendation function for the model (model is located in [aibolit/binary_files/model.pkl](https://github.com/yegor256/aibolit/blob/master/aibolit/binary_files/model.pkl). 
-The model finds a pattern which contribution is the largest to the Cyclomatic Complexity. 
-If anything is found, you will see all recommendations for the mentioned patterns. 
-You can see the list of all patterns in [Patterns.md](https://github.com/yegor256/aibolit/blob/master/PATTERNS.md).
-The output of recommendation will be redirected to the stdout. 
+It will run recommendation function for the model (model is located in [aibolit/binary_files/model.pkl](https://github.com/cqfn/aibolit/blob/master/aibolit/binary_files/model.pkl).
+The model finds a pattern which contribution is the largest to the Cyclomatic Complexity.
+If anything is found, you will see all recommendations for the mentioned patterns.
+You can see the list of all patterns in [Patterns.md](https://github.com/cqfn/aibolit/blob/master/PATTERNS.md).
+The output of recommendation will be redirected to the stdout.
 If the program has the `0` exit code, it means that all analyzed files do not have any issues.
 If the program has the `1` exit code, it means that at least 1 analyzed file has an issue.
 If the program has the `2` exit code, it means that program crash occurred.
@@ -73,7 +71,7 @@ Show all patterns
 /mnt/d/src/java/Configuration.java[3261]: Partial synchronized (P14: 0.228 4/4)
 /mnt/d/src/java/Configuration.java[3727]: Partial synchronized (P14: 0.228 4/4)
 /mnt/d/src/java/Configuration.java[3956]: Partial synchronized (P14: 0.228 4/4)
-/mnt/d/src/java/ErrorExample.java: error when calculating patterns: Can't count P1 metric: 
+/mnt/d/src/java/ErrorExample.java: error when calculating patterns: Can't count P1 metric:
 Total score: 127.67642529949538
 ```
 
@@ -106,7 +104,7 @@ Show all patterns
 /mnt/d/src/java/Configuration.java[3844]: Var in the middle (P21: 30.95612931128819 1/4)
 /mnt/d/src/java/Configuration.java[3848]: Var in the middle (P21: 30.95612931128819 1/4)
 /mnt/d/src/java/Configuration.java[3956]: Partial synchronized (P14: 0.228 4/4)
-/mnt/d/src/java/ErrorExample.java: error when calculating patterns: Can't count P1 metric: 
+/mnt/d/src/java/ErrorExample.java: error when calculating patterns: Can't count P1 metric:
 /mnt/d/src/java/MavenSlice.java: your code is perfect in aibolit's opinion
 Total score: 127.67642529949538
 ```
@@ -172,8 +170,8 @@ You can also choose xml format. It will have the same format as `text` mode, but
 </report>
 
 ```
-The score is the relative importance of the pattern (there is no range for it). 
-The larger score is, the most important pattern is. 
+The score is the relative importance of the pattern (there is no range for it).
+The larger score is, the most important pattern is.
 E.g., if you have several patterns, first you need to fix the pattern with the score 5.45:
 
 ```
@@ -219,37 +217,38 @@ You can get full report with `--full` command, then all patterns will be include
 $ aibolit recommend --folder src/java --full
 ```
 
-You can exclude folder with `--exclude` command. The last parameter is the folder to exclude, 
+You can exclude folder with `--exclude` command. The last parameter is the folder to exclude,
 the rest of them are glob patterns.
 ```bash
 $ aibolit recommend --folder src/java --exclude=**/*Test*.java --exclude=**/*Impl*.java --exclude=/mnt/d/src/java/tests
 ```
 
-If you need help, run 
+If you need help, run
 
 ```bash
 $ aibolit recommend --help
 ```
 
-### Train command
+## How to retrain it?
+
 `Train` command does the following:
 
  - Calculates patterns and metrics
  - Creates a dataset
- - Trains model and save it 
- 
+ - Trains model and save it
+
  Train works only with cloned git repository.
- 1. Clone aibolit repository 
+ 1. Clone aibolit repository
  2. Go to `cloned_aibolit_path`
  3. Run `pip install .`
  4. Set env variable `export HOME_AIBOLIT=cloned_aibolit_path` (example for Linux).
  5. If you need to set up own directory where model will be saved, set up also `SAVE_MODEL_FOLDER` environment variable.
  Otherwise model will be saved into `cloned_aibolit_path/aibolit/binary_files/model.pkl`
  6. If you need to set up own folder with Java files, use `--java_folder parameter`, the default value will be `scripts/target/01` of aibolit cloned repo
  7. You need to install Java 13 and Maven
- 
+
  Or you can use our docker image (link will be soon here)
- 
+
  Run train pipeline:
 
 ```bash
@@ -299,5 +298,5 @@ Using Docker recommendation pipeline
 $ docker run --rm -it \
   -v <absolute_path_to_folder_with_classes>:/in \
   -v <absolute_path_to_out_dir>:/out \
-  yegor256/aibolit-image
+  cqfn/aibolit-image
 ```

aibolit/__main__.py

@@ -33,10 +33,13 @@
 import sys
 import time
 import traceback
+from collections import defaultdict
 from os import scandir
 from pathlib import Path
 from sys import stdout
-from typing import List
+from typing import List, Any, Dict, Tuple
+
+import javalang
 import numpy as np  # type: ignore
 import requests
 from lxml import etree  # type: ignore
@@ -45,7 +48,6 @@
 from aibolit import __version__
 from aibolit.config import Config
 from aibolit.ml_pipeline.ml_pipeline import train_process, collect_dataset
-from aibolit.model.model import TwoFoldRankingModel, Dataset  # type: ignore  # noqa: F401
 
 dir_path = os.path.dirname(os.path.realpath(__file__))
 
@@ -159,6 +161,90 @@ def __count_value(value_dict, input_params, code_lines_dict, java_file: str, is_
         )
 
 
+def flatten(l):
+    return [item for sublist in l for item in sublist]
+
+
+def add_pattern_if_ignored(
+        dct: Dict[str, Any],
+        pattern_item: Dict[Any, Any],
+        results_list: List[Any]) -> None:
+    """ If pattern code is not ignored, add it to the result list
+
+    :param dct: dict, where key is pattern, value is list of lines range to ignore
+    :param pattern_item: pattern dict which was get after `inference` function
+    :param results_list: result list to add
+
+    """
+    ignored_lines = dct.get(pattern_item['pattern_code'])
+    if ignored_lines:
+        for place in ignored_lines:
+            # get lines range of ignored code
+            start_line_to_ignore = place[0]
+            end_line_to_ignore = place[1]
+            new_code_lines = []
+            for line in pattern_item['code_lines']:
+                if (line >= start_line_to_ignore) and (line <= end_line_to_ignore):
+                    continue
+                else:
+                    new_code_lines.append(line)
+            pattern_item['code_lines'] = new_code_lines
+            if len(pattern_item['code_lines']) > 0:
+                results_list.append(pattern_item)
+    else:
+        results_list.append(pattern_item)
+
+
+def find_annotation_by_node_type(
+        tree: javalang.tree.CompilationUnit,
+        node_type) -> Dict[Any, Any]:
+    """Search nodes with annotations.
+
+    :param tree: javalang.tree
+    :param node_type: Node type of javalang.tree
+    :return
+    dict with annotations, where key is node, value is list of string annotations;
+    """
+    annonations = defaultdict(list)
+    for _, node in tree.filter(node_type):
+        if node.annotations:
+            for a in node.annotations:
+                if hasattr(a.element, 'value'):
+                    if 'aibolit' in a.element.value:
+                        annonations[node].append(
+                            a.element.value.split('.')[1].rstrip('\"')
+                        )
+                elif hasattr(a.element, 'values'):
+                    for j in a.element.values:
+                        if 'aibolit' in j.value:
+                            annonations[node].append(
+                                j.value.split('.')[1].rstrip('\"')
+                            )
+    return annonations
+
+
+def find_start_and_end_lines(node) -> Tuple[int, int]:
+    max_line = node.position.line
+
+    def traverse(node):
+        if hasattr(node, 'children'):
+            for child in node.children:
+                if isinstance(child, list) and (len(child) > 0):
+                    for item in child:
+                        traverse(item)
+                else:
+                    if hasattr(child, '_position'):
+                        nonlocal max_line
+                        if child._position.line > max_line:
+                            max_line = child._position.line
+                            return
+        else:
+            return
+
+    traverse(node)
+    return node.position.line, max_line
+
+
 def calculate_patterns_and_metrics(file, args):
     code_lines_dict = input_params = {}  # type: ignore
     error_string = None
@@ -586,12 +672,11 @@ def handle_exclude_command_line(args):
 
 
 def format_converter_for_pattern(results, sorted_by=None):
-    """Reformat data where data are sorted by patterns importance
+    """
+    Reformat data where data are sorted by patterns importance
     (it is already sorted in the input).
-    Then lines are sorted in ascending order."""
-
-    def flatten(l):
-        return [item for sublist in l for item in sublist]
+    Then lines are sorted in ascending order.
+    """
 
     for file in results:
         items = file.get('results')

aibolit/ast_framework/ast.py

@@ -35,6 +35,8 @@
 
 MemberReferenceParams = namedtuple('MemberReferenceParams', ('object_name', 'member_name', 'unary_operator'))
 
+BinaryOperationParams = namedtuple('BinaryOperationParams', ('operation', 'left_side', 'right_side'))
+
 
 class AST:
     _NODE_SKIPED = -1
@@ -179,6 +181,11 @@ def get_member_reference_params(self, member_reference_node: int) -> MemberRefer
 
         return member_reference_params
 
+    def get_binary_operation_params(self, binary_operation_node: int) -> BinaryOperationParams:
+        assert(self.get_type(binary_operation_node) == ASTNodeType.BINARY_OPERATION)
+        operation_node, left_side_node, right_side_node = self.tree.succ[binary_operation_node]
+        return BinaryOperationParams(self.get_attr(operation_node, 'string'), left_side_node, right_side_node)
+
     @staticmethod
     def _build_from_javalang(tree: DiGraph, javalang_node: Node) -> int:
         node_index = len(tree) + 1