Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge features, fixes, and rewrites performed by EUMETSAT (towards SIFT 2.0) #341

Merged
merged 1,337 commits into from
May 14, 2023
Merged

Merge features, fixes, and rewrites performed by EUMETSAT (towards SIFT 2.0) #341

merged 1,337 commits into from
May 14, 2023

Conversation

djhoese
Copy link
Member

@djhoese djhoese commented May 9, 2023

Replacement of #335

This is a giant PR that merges everything from the EUMETSAT upstream master branch. This is 2+ years of work I think.

There are a few flake8 issues and sphinx website issues caused by the more recent versions of flake8 and sphinx that need to be cleaned up. There are also one or two more merge requests waiting on the EUM gitlab side of things that I'll need to merge into this before this is merged.

Note that there will need to be more PRs/changes after this is merged, but this is a first step at getting these code bases back together.

CC @ameraner

askalex and others added 30 commits January 3, 2023 04:11
@askalex
Resolve mypy issues wrt to info type 'frozendict' (layer_item.py, lay…
a6ab7ad 
…er_model.py, product_dataset.py)

Get rid of

uwsift/model/layer_model.py:504: error: Argument 2 to "LayerItem" has incompatible type "Dict[Info, Kind]"; expected "frozendict"  [arg-type]

uwsift/model/layer_model.py:594: error: Argument 4 to "add_multichannel_dataset" of "LayerItem" has incompatible type "List[Dict[Any, Any]]"; expected "List[frozendict]"  [arg-type]

by fine tuning type annotations
@askalex
Resolve mypy issues in 'open_file_wizard.py'
1d70b0b 
Get rid of

uwsift/view/open_file_wizard.py:577: error: Need type annotation for "scenes" (hint: "scenes: Dict[<type>, <type>] = ...")  [var-annotated]
uwsift/view/open_file_wizard.py:579: error: Need type annotation for "known_files" (hint: "known_files: Set[<type>] = ...")  [var-annotated]
uwsift/view/open_file_wizard.py:661: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
uwsift/view/open_file_wizard.py:749: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]

by adding type annotations
@askalex
Resolve mypy issues in 'rgb_config.py'
a973365 
Get rid of

uwsift/view/rgb_config.py:38: error: Incompatible types in assignment (expression has type "None", variable has type "List[Tuple[float, float]]")  [assignment]
uwsift/view/rgb_config.py:44: error: List item 0 has incompatible type "Tuple[None, None]"; expected "Tuple[float, float]"  [list-item]
uwsift/view/rgb_config.py:44: error: List item 1 has incompatible type "Tuple[None, None]"; expected "Tuple[float, float]"  [list-item]
uwsift/view/rgb_config.py:44: error: List item 2 has incompatible type "Tuple[None, None]"; expected "Tuple[float, float]"  [list-item]
uwsift/view/rgb_config.py:45: error: Need type annotation for "_layer_uuids" (hint: "_layer_uuids: List[<type>] = ...")  [var-annotated]
uwsift/view/rgb_config.py:220: error: "None" has no attribute "input_layer_ids"  [attr-defined]
uwsift/view/rgb_config.py:223: error: Item "None" of "Optional[LayerItem]" has no attribute "info"  [union-attr]
uwsift/view/rgb_config.py:233: error: Incompatible default for argument "n" (default has type "None", argument has type "float")  [assignment]
uwsift/view/rgb_config.py:233: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/view/rgb_config.py:233: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/view/rgb_config.py:233: error: Incompatible default for argument "x" (default has type "None", argument has type "float")  [assignment]
uwsift/view/rgb_config.py:267: error: Incompatible default for argument "color" (default has type "None", argument has type "str")  [assignment]
uwsift/view/rgb_config.py:267: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/view/rgb_config.py:267: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/view/rgb_config.py:308: error: Incompatible types in assignment (expression has type "Optional[Recipe]", variable has type "None")  [assignment]
uwsift/view/rgb_config.py:348: error: Incompatible types in assignment (expression has type "Tuple[None, None]", target has type "Tuple[float, float]")  [assignment]
uwsift/view/rgb_config.py:368: error: Item "None" of "Optional[LayerItem]" has no attribute "valid_range"  [union-attr]
uwsift/view/rgb_config.py:370: error: Item "None" of "Optional[LayerItem]" has no attribute "get_actual_range_from_layer"  [union-attr]
uwsift/view/rgb_config.py:439: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]

by
- making class variables member variables to avoid any need to
  initialize them with None
- make optional keywoard arguments mandatory arguments to avoid the need
  to type them as Optional[<type>]
- fine tuning type annotations
- calming down mypy with assertions that states explicityly, that
  variables in question cannot be unset
@askalex
Resolve mypy issues in 'algebraic_config.py' files
b06cfb7 
Get rid of

uwsift/view/algebraic_config.py:43: error: Need type annotation for "_layer_uuids" (hint: "_layer_uuids: List[<type>] = ...")  [var-annotated]
uwsift/view/algebraic_config.py:111: error: Incompatible types in assignment (expression has type "Optional[Recipe]", variable has type "None")  [assignment]

by adding type annotations
@askalex
Resolve mypy issues in 'probes.py'
724ac59 
Get rid of

uwsift/view/probes.py:43: error: Name "NavigationToolbar" already defined (possibly by an import)  [no-redef]
uwsift/view/probes.py:125: error: Need type annotation for "point_probes" (hint: "point_probes: Dict[<type>, <type>] = ...")  [var-annotated]
uwsift/view/probes.py:128: error: Need type annotation for "graphs" (hint: "graphs: List[<type>] = ...")  [var-annotated]

by
- renaming locally defined class NavigationToolbar to
  CustomNavigationToolbar
- adding requested type annotations
@askalex
Resolve mypy issues in 'tile_calculator.py'
4a9135e 
Get rid of

uwsift/view/tile_calculator.py:567: error: Value of type "None" is not indexable  [index]
uwsift/view/tile_calculator.py:568: error: Value of type "None" is not indexable  [index]
uwsift/view/tile_calculator.py:573: error: Value of type "None" is not indexable  [index]
uwsift/view/tile_calculator.py:574: error: Value of type "None" is not indexable  [index]
uwsift/view/tile_calculator.py:575: error: Value of type "None" is not indexable  [index]
uwsift/view/tile_calculator.py:576: error: Value of type "None" is not indexable  [index]

by removing class variables which would be shadowed by member variables
anyway to avoid any need to initialize them with None
@askalex
Resolve mypy issues in 'time_matcher.py'
1bc5ae1 
Get rid of

uwsift/control/time_matcher.py:9: error: Incompatible default for argument "policy" (default has type "None", argument has type "Callable[..., Any]")  [assignment]
uwsift/control/time_matcher.py:9: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/control/time_matcher.py:9: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase

by fine tuning type annotations
@askalex
Resolve mypy issues in 'time_transformer_policies.py'
876ba79 
Get rid of

uwsift/control/time_transformer_policies.py:80: error: Incompatible types in assignment (expression has type "LayerItem", variable has type "None")  [assignment]
uwsift/control/time_transformer_policies.py:81: error: "None" has no attribute "timeline"  [attr-defined]
uwsift/control/time_transformer_policies.py:102: error: Incompatible types in assignment (expression has type "List[datetime]", variable has type "None")  [assignment]
uwsift/control/time_transformer_policies.py:172: error: Incompatible return value type (got "None", expected "datetime")  [return-value]

by
- adding type annotations
- make a method deal correctly when None is passed for one argument
@askalex
Resolve mypy issues in 'time_manager.py'
d44d5e7 
Get rid of

uwsift/model/time_manager.py:61: error: Attribute "_layer_model" already defined on line 41  [no-redef]
uwsift/model/time_manager.py:93: error: Item "None" of "Optional[TimeTransformer]" has no attribute "step"  [union-attr]
uwsift/model/time_manager.py:155: error: Item "None" of "Optional[LayerModel]" has no attribute "get_dynamic_layers"  [union-attr]
uwsift/model/time_manager.py:178: error: Item "None" of "Optional[TimeTransformer]" has no attribute "update_current_timebase"  [union-attr]
uwsift/model/time_manager.py:182: error: Item "None" of "Optional[TimeTransformer]" has no attribute "t_sim"  [union-attr]
uwsift/model/time_manager.py:185: error: Item "None" of "Optional[TimeTransformer]" has no attribute "t_sim"  [union-attr]

by
- calming down mypy with assertions that states explicityly, that
  variables in question cannot be unset
- removing a redundant type annotation
@askalex
Resolve mypy issues in 'qml_utils.py'
e825d3b 
Get rid of

uwsift/control/qml_utils.py:87: error: Name "layerModel" already defined on line 83  [no-redef]
uwsift/control/qml_utils.py:96: error: Name "convFuncModel" already defined on line 92  [no-redef]
uwsift/control/qml_utils.py:132: error: Name "layerToDisplay" already defined on line 127  [no-redef]
uwsift/control/qml_utils.py:144: error: Name "dateToDisplay" already defined on line 137  [no-redef]
uwsift/control/qml_utils.py:221: error: Property "timestamps" defined in "TimebaseModel" is read-only  [misc]
uwsift/control/qml_utils.py:222: error: Cannot assign to a method  [assignment]
uwsift/control/qml_utils.py:222: error: Incompatible types in assignment (expression has type "None", variable has type "Callable[..., Any]")  [assignment]
uwsift/control/qml_utils.py:243: error: Name "currentTimestamp" already defined on line 239  [no-redef]
uwsift/control/qml_utils.py:261: error: Name "timestamps" already defined on line 248  [no-redef]
uwsift/control/qml_utils.py:261: error: "Callable[[TimebaseModel], Any]" has no attribute "setter"  [attr-defined]

by
- ignoring false positives for pyqtProperty setters, see
  python/mypy#9911
- Avoiding to initialize variable with None which can also be
  initialized with something more useful right away.
@askalex
Resolve mypy issues in 'logger.py'
f9f284d 
Get rid of

uwsift/util/logger.py:30: error: Argument 1 to "__configure_handler" has incompatible type "Optional[str]"; expected "str"  [arg-type]
uwsift/util/logger.py:60: error: Name "handler" already defined on line 45  [no-redef]
uwsift/util/logger.py:62: error: Name "err_msg" already defined on line 51  [no-redef]

by
- fine tuning type annotations
- annotiating a variable with a super type to allow it to hold two
  different derived types
- removing redundant type annotations
@askalex
Resolve mypy issues in 'document.py'
fd1d666 
Get rid of

uwsift/model/document.py:71: error: Incompatible types in assignment (expression has type "None", variable has type "str")  [assignment]
uwsift/model/document.py:72: error: Incompatible types in assignment (expression has type "None", variable has type "TaskQueue")  [assignment]
uwsift/model/document.py:73: error: Incompatible types in assignment (expression has type "None", variable has type "BaseWorkspace")  [assignment]
uwsift/model/document.py:99: error: Need type annotation for "_info_by_uuid" (hint: "_info_by_uuid: Dict[<type>, <type>] = ...")  [var-annotated]
uwsift/model/document.py:204: error: Argument 1 to "dict" has incompatible type "Optional[frozendict]"; expected "SupportsKeysAndGetItem[Any, Any]"  [arg-type]
uwsift/model/document.py:247: error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]

by
- making class variables member variables to avoid any need to
  initialize them with None
- fine tuning type annotations
- revising the way a dict is generated as a copy of a
  Optional[frozendict]

and of

uwsift/model/document.py:282: error: Too many arguments for "find_merge_target" of "BaseWorkspace"  [call-arg]

by fixing the discrepancy of the signature of the
BaseWorkspace.find_merge_target() method compared to the correct
SimpleWorkspace.find_merge_target() one.
@askalex
Resolve mypy issues in 'importer.py'
aa473ed 
Get rid of

-uwsift/workspace/importer.py:174: error: "Guidebook" has no attribute "collect_info"  [attr-defined]
-uwsift/workspace/importer.py:178: error: Argument 2 to "get_default_colormap" has incompatible type "Guidebook"; expected "ABI_AHI_Guidebook"  [arg-type]
-uwsift/workspace/importer.py:181: error: "Guidebook" has no attribute "valid_range"  [attr-defined]
-uwsift/workspace/importer.py:185: error: "Guidebook" has no attribute "_default_display_time"  [attr-defined]
-uwsift/workspace/importer.py:187: error: "Guidebook" has no attribute "_default_display_name"  [attr-defined]
-uwsift/workspace/importer.py:256: error: Incompatible types in assignment (expression has type "None", variable has type "str")  [assignment]
uwsift/workspace/importer.py:313: error: "aImporter" gets multiple values for keyword argument "workspace_cwd"  [misc]
uwsift/workspace/importer.py:407: error: Return value expected  [return-value]
uwsift/workspace/importer.py:516: error: Incompatible types in assignment (expression has type "dict_values[Any, Any]", variable has type "List[Any]")  [assignment]
uwsift/workspace/importer.py:800: error: Incompatible default for argument "name_prefix" (default has type "None", argument has type "str")  [assignment]
uwsift/workspace/importer.py:800: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/importer.py:800: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/importer.py:833: error: "Guidebook" has no attribute "_default_display_name"  [attr-defined]
uwsift/workspace/importer.py:868: error: Incompatible return value type (got "Dict[Any, Set[Any]]", expected "Dict[str, List[str]]")  [return-value]
uwsift/workspace/importer.py:882: error: Unsupported left operand type for & ("List[str]")  [operator]
uwsift/workspace/importer.py:1214: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
uwsift/workspace/importer.py:1288: error: Need type annotation for "visited_segments" (hint: "visited_segments: List[<type>] = ...")  [var-annotated]

by
- tuning type annotations
- removing class variables which would be shadowed by member variables
  anyway to avoid any need to initialize them with None
- casting a dict.values() result to a list
- replacing references to the base class 'Guidebook' by those to its
  (currently) only sub-class 'ABI_AHI_Guidebook'. This provisionally
  hides the issue, that the base class has not the full interface of the
  sub-class.
  TODO: clean up Guidebook class hierarchy
@askalex
Resolve mypy issues in 'collector.py'
c0630b1 
Get rid of

uwsift/workspace/collector.py:65: error: Incompatible types in assignment (expression has type "None", variable has type "BaseWorkspace")  [assignment]
uwsift/workspace/collector.py:66: error: Incompatible types in assignment (expression has type "None", variable has type "List[str]")  [assignment]
uwsift/workspace/collector.py:67: error: Incompatible types in assignment (expression has type "None", variable has type "Mapping[str, datetime]")  [assignment]
uwsift/workspace/collector.py:68: error: Incompatible types in assignment (expression has type "None", variable has type "str")  [assignment]
uwsift/workspace/collector.py:69: error: Incompatible types in assignment (expression has type "None", variable has type "bool")  [assignment]
uwsift/workspace/collector.py:70: error: Incompatible types in assignment (expression has type "None", variable has type "List[str]")  [assignment]
uwsift/workspace/collector.py:71: error: Incompatible types in assignment (expression has type "None", variable has type "List[str]")  [assignment]
uwsift/workspace/collector.py:104: error: Incompatible default for argument "dirs" (default has type "None", argument has type "Iterable[str]")  [assignment]
uwsift/workspace/collector.py:104: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/collector.py:104: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/collector.py:136: error: Incompatible types in assignment (expression has type "Union[BaseWorkspace, _workspace_test_proxy]", variable has type "BaseWorkspace")  [assignment]

by
- fixing type annotations
- removing class variables which would be shadowed by member variables
  anyway to avoid any need to initialize them with None
- moving ResourceSearchPathCollector.__init__() to where it belongs (top
  of the class) to make sure typed variable definitions in __init__ are
  seen before any other references to the same variables
@askalex
Resolve mypy issues in 'metadatabase.py'
405204f 
Get rid of

uwsift/workspace/metadatabase.py:82: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:82: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:82: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:179: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:179: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:179: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:472: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:472: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:472: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:486: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:486: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:486: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:498: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:498: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:498: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:769: error: Variable "uwsift.workspace.metadatabase.Base" is not valid as a type  [valid-type]
uwsift/workspace/metadatabase.py:769: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
uwsift/workspace/metadatabase.py:769: error: Invalid base class "Base"  [misc]
uwsift/workspace/metadatabase.py:848: error: Value of type "None" is not indexable  [index]
uwsift/workspace/metadatabase.py:848: error: Unsupported target for indexed assignment ("None")  [index]

by
- hiding class declarations from mypy for classes which derive from the
  SQLAlchemy generated 'Base=declarative_base()': even adding
  'sqlalchemy' as mypy 'additional_dependency' in
  .pre-commit-config.yaml didn't help to resolve the issue described
  also here: https://docs.sqlalchemy.org/en/20/orm/extensions/mypy.html
- calming down mypy with an assertion that states explicityly, that a
  variable in question cannot be unset
@askalex
Prepare for Workspace classes interface cleanup
0ba28b3 
Only CachingWorkspace needs the caching related interface. Several
client methods are (and must) only be called, when the instantiated
Workspace is a CachingWorkspace (which is the case, with the
configuration setting "storage.use_inventory_db: True").  Make this
explicit by assertions which check the type of the Workspace.

This secures it to remove the methods which are specific to the caching
use case from BaseWorkspace and SimpleWorkspace, which is foreseen for
the next commit.
@askalex
Resolve mypy issues in '[{caching,simple}_]workspace.py'
fc24c4b 
Get rid of

uwsift/workspace/workspace.py:131: error: Incompatible types in assignment (expression has type "Optional[Dict[Any, Any]]", variable has type "Dict[Any, Any]")  [assignment]
uwsift/workspace/workspace.py:255: error: Incompatible types in assignment (expression has type "None", variable has type "Mapping[int, ActiveContent]")  [assignment]
uwsift/workspace/workspace.py:256: error: Incompatible types in assignment (expression has type "None", variable has type "Metadatabase")  [assignment]
uwsift/workspace/workspace.py:271: error: Incompatible types in assignment (expression has type "None", variable has type "Mapping[UUID, Flags]")  [assignment]
uwsift/workspace/workspace.py:294: error: Incompatible default for argument "directory_path" (default has type "None", argument has type "str")  [assignment]
uwsift/workspace/workspace.py:294: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/workspace.py:294: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/workspace.py:306: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/workspace.py:312: error: Cannot determine type of "TheWorkspace"  [has-type]
uwsift/workspace/workspace.py:321: error: Incompatible types in assignment (expression has type "bool", variable has type "None")  [assignment]
uwsift/workspace/workspace.py:361: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/workspace.py:361: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/workspace.py:361: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/workspace.py:361: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/workspace.py:367: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/workspace.py:367: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/workspace.py:367: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/workspace.py:367: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/workspace.py:469: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/workspace.py:469: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/workspace.py:469: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/workspace.py:470: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/workspace.py:470: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/workspace.py:470: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/workspace.py:667: error: Incompatible return value type (got "Tuple[None, None]", expected "Tuple[int, int]")  [return-value]
uwsift/workspace/workspace.py:682: error: Incompatible return value type (got "Tuple[None, None]", expected "Tuple[int, int]")  [return-value]
uwsift/workspace/workspace.py:753: error: Value of type "Optional[Any]" is not indexable  [index]
uwsift/workspace/workspace.py:811: error: Incompatible types in assignment (expression has type "Optional[ActiveContent]", variable has type "ActiveContent")  [assignment]
uwsift/workspace/workspace.py:816: error: Incompatible return value type (got "Optional[Any]", expected "Dict[Any, Any]")  [return-value]
uwsift/workspace/workspace.py:825: error: Incompatible types in assignment (expression has type "Optional[ActiveContent]", variable has type "ActiveContent")  [assignment]
uwsift/workspace/simple_workspace.py:34: error: Incompatible default for argument "directory_path" (default has type "None", argument has type "str")  [assignment]
uwsift/workspace/simple_workspace.py:34: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/simple_workspace.py:34: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/simple_workspace.py:49: error: Incompatible return value type (got "None", expected "Metadatabase")  [return-value]
uwsift/workspace/simple_workspace.py:62: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:81: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:84: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:86: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:92: error: Return type "Optional[Product]" of "_product_with_uuid" incompatible with return type "Product" in supertype "BaseWorkspace"  [override]
uwsift/workspace/simple_workspace.py:96: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/simple_workspace.py:96: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/simple_workspace.py:96: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/simple_workspace.py:96: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/simple_workspace.py:101: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/simple_workspace.py:101: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/simple_workspace.py:101: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/simple_workspace.py:101: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/simple_workspace.py:166: error: Incompatible return value type (got "None", expected "Dict[Any, Any]")  [return-value]
uwsift/workspace/simple_workspace.py:168: error: Missing return statement  [empty-body]
uwsift/workspace/simple_workspace.py:219: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:242: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/simple_workspace.py:242: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/simple_workspace.py:242: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/simple_workspace.py:243: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/simple_workspace.py:243: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/simple_workspace.py:243: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/simple_workspace.py:268: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:372: error: Unsupported right operand type for in ("Info")  [operator]
uwsift/workspace/simple_workspace.py:374: error: No overload variant of "dict" matches argument type "Info"  [call-overload]
uwsift/workspace/simple_workspace.py:374: note: Possible overload variants:
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self) -> Dict[_KT, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, SupportsKeysAndGetItem[_KT, _VT], /) -> Dict[_KT, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, SupportsKeysAndGetItem[str, _VT], /, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, Iterable[Tuple[_KT, _VT]], /) -> Dict[_KT, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, Iterable[Tuple[str, _VT]], /, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/simple_workspace.py:374: note:     def [_KT, _VT] dict(self, Iterable[List[str]], /) -> Dict[str, str]
uwsift/workspace/simple_workspace.py:386: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/simple_workspace.py:397: error: Value of type "Info" is not indexable  [index]
uwsift/workspace/simple_workspace.py:398: error: Value of type "Info" is not indexable  [index]
uwsift/workspace/simple_workspace.py:422: error: Argument 1 to "_deactivate_content_for_product" of "SimpleWorkspace" has incompatible type "Optional[Product]"; expected "Product"  [arg-type]
uwsift/workspace/caching_workspace.py:216: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/caching_workspace.py:228: error: Unsupported target for indexed assignment ("Mapping[int, ActiveContent]")  [index]
uwsift/workspace/caching_workspace.py:228: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/caching_workspace.py:251: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/caching_workspace.py:251: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/caching_workspace.py:251: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/caching_workspace.py:251: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/caching_workspace.py:270: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/caching_workspace.py:270: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/caching_workspace.py:270: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/caching_workspace.py:270: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/caching_workspace.py:536: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/caching_workspace.py:550: error: Incompatible default for argument "uuid" (default has type "None", argument has type "UUID")  [assignment]
uwsift/workspace/caching_workspace.py:550: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/caching_workspace.py:550: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/caching_workspace.py:551: error: Incompatible default for argument "prod" (default has type "None", argument has type "Product")  [assignment]
uwsift/workspace/caching_workspace.py:551: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
uwsift/workspace/caching_workspace.py:551: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
uwsift/workspace/caching_workspace.py:570: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/caching_workspace.py:617: error: Unsupported right operand type for in ("Info")  [operator]
uwsift/workspace/caching_workspace.py:619: error: No overload variant of "dict" matches argument type "Info"  [call-overload]
uwsift/workspace/caching_workspace.py:619: note: Possible overload variants:
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self) -> Dict[_KT, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, SupportsKeysAndGetItem[_KT, _VT], /) -> Dict[_KT, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, SupportsKeysAndGetItem[str, _VT], /, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, Iterable[Tuple[_KT, _VT]], /) -> Dict[_KT, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, Iterable[Tuple[str, _VT]], /, **kwargs: _VT) -> Dict[str, _VT]
uwsift/workspace/caching_workspace.py:619: note:     def [_KT, _VT] dict(self, Iterable[List[str]], /) -> Dict[str, str]
uwsift/workspace/caching_workspace.py:631: error: Cannot determine type of "cache_dir"  [has-type]
uwsift/workspace/caching_workspace.py:641: error: Value of type "Info" is not indexable  [index]
uwsift/workspace/caching_workspace.py:642: error: Value of type "Info" is not indexable  [index]
uwsift/workspace/caching_workspace.py:720: error: "Mapping[int, ActiveContent]" has no attribute "pop"  [attr-defined]

by
- making some class variables of BaseWorkspace member variables of
  CachingWorkspace, thus move them where they belong and avoid the need
  to initialize them with None
- removing methods from {Base,Simple}Workspace, which are only
  implemented for CachingWorkspace and must not called for the
  SimpleWorkspace anyway
- tuning type annotations
- fixing errorneous type annotations using 'Info' (which is an Enum defined
  in this package) as a type where 'Mapping' is correct
- avoiding default value None for keyword args where possible
- calming down mypy with assertions that state explicityly, that some
  variables in question cannot be unset
- fixing a bug: {}.update({key: value}) does not return anything, i.e.,
  attrs = attrs.update(...) is plain wrong
- avoiding aliasing member variables by local variables
@askalex
Clean up {Base,Caching}Workspace implementation
d3057e8 
Remove class variables that will later be overshadowed by member
variables of the same name anyway when the latter are initialized - the
old code from early SIFT development is obviously the result of a
misconception about class and member variables in Python.

Clean up the initialization of BaseWorkspace._queue: the variable is
used in BaseWorkspace methods, but only initialized in the derived class
CachingWorkspace. This is improved by passing through the initialization
given to CachingWorkspace.__init__() to BaseWorkspace.__init__().

The matter was spotted during mypy issue resolution but it belongs
better to #89 in terms of its concern.
@askalex
Resolve mypy issue in 'organize_data_topics.py'
5a85537 
Get rid of

uwsift/project/organize_data_topics.py:49: error: Need type annotation for "guam_cases" (hint: "guam_cases: Dict[<type>, <type>] = ...")  [var-annotated]

by adding a type annotation
@askalex
Run black, isort on etc/.../fill_dir_periodically.py
3dc1abf 
to reformat the file.

This commit is only in preparation for the next commit dealing with mypy
issues.
@askalex
Resolve mypy issues in '**/fill_dir_periodically.py'
8e4e622 
Get rid of

etc/mtgsift-dist-template/etc/tests/fill_dir_periodically.py:172: error: Cannot determine type of "tmp_path"  [has-type]
uwsift/tests/control/fill_dir_periodically.py:205: error: Argument 2 to "fill_dir_periodically_seviri" has incompatible type "Path"; expected "str"  [arg-type]
uwsift/tests/control/fill_dir_periodically.py:207: error: Argument 2 to "fill_dir_periodically_fci" has incompatible type "Path"; expected "str"  [arg-type]
@askalex
Resolve mypy issues in 'heap_analyzer.py'
550fa2e 
Get rid of

uwsift/util/heap_analyzer.py:21: error: Need type annotation for "_allocations"  [var-annotated]
uwsift/util/heap_analyzer.py:32: error: Item "None" of "Optional[Match[str]]" has no attribute "group"  [union-attr]

by
- adding type annotation
- dealing correctly with the case that a variable might become None
@askalex
Resolve mypy issues in 'watchdog.py'
c9d3450 
Get rid of

uwsift/util/watchdog.py:115: error: Incompatible types in assignment (expression has type "int", variable has type "str")  [assignment]
uwsift/util/watchdog.py:121: error: Incompatible return value type (got "str", expected "int")  [return-value]
uwsift/util/watchdog.py:123: error: Incompatible return value type (got "str", expected "int")  [return-value]

by
- fixing type annotations
- avoiding the reuse of a variable with a different type
@askalex
Resolve mypy issues in 'doc/.../conf.py'
ed5ddaf 
Get rid of

doc/source/conf.py:126: error: Need type annotation for "exclude_patterns" (hint: "exclude_patterns: List[<type>] = ...")  [var-annotated]
doc/source/conf.py:256: error: Need type annotation for "latex_elements" (hint: "latex_elements: Dict[<type>, <type>] = ...")  [var-annotated]

by adding type annotations.
@askalex
Resolve mypy issue in 'setup.py'
33e68b9 
Get rid of

setup.py:57: error: Item "None" of "Optional[Match[str]]" has no attribute "groupdict"  [union-attr]

by fixing the implementation
@askalex
Merge branch 'feature/wp5/142-solve_mypy_issues' into feature/wp5/142…
2777925 
…-bandit_and_mypy_issues
@askalex
Reenable bandit pre-commit hook
f500a73
@askalex
Resolve bandit [B101:assert_used] issues in all files
8a9d3c3 
Get rid of
- [B101:assert_used] Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

All assert statements are reviewed and either:
- excluded from being reported (by adding '# nosec B101') when they
  don't influence normal control flow and have no side effects but add
  value for maintenance and understanding of the code [most occurences]
- converted to an if statement including raising a ValueError exception
  in the case of former assertion violation when the assertion
  directly or indirectly validates user input (configuration is
  considered as user input) [only one occurrence]
- removed if they are considered useless [a few occurrences]
@askalex
Suppress [B108:hardcoded_tmp_directory] warnings in 'collector.py'
0a36906 
... namely two reports of
[B108:hardcoded_tmp_directory] Probable insecure usage of temp file/directory.
for
Location: uwsift/workspace/collector.py:44

The concerned code is only executed, when a test is run via the 'main'
function of that file and never during normal SIFT operation.
@askalex
Suppress [B102:exec_used] warning in 'workspace.py'
d01f092 
Suppress reporting two findings of
[B102:exec_used] Use of exec detected.
at
Location: uwsift/workspace/workspace.py:551:8
Location: uwsift/workspace/workspace.py:555:8

Execution of a user provided code snippet is mandatory to implement the
algebraics feature.

It is not an attack vector because to change the executed code one needs
access to the SIFT GUI.

A careless user could possibly make malicious code run, but only with
his privileges.
djhoese and others added 27 commits May 9, 2023 20:55
@djhoese
Update versions of python used in CI
a7e69b1
@ameraner
set APPLICATION_AUTHOR to False
a13d1bc
@ameraner
Merge branch 'update_author_names' into 'master'
32c0c86 
Update Author Name

See merge request webservices/mtg-sift!160
@ameraner
remove unused settings.version
70a6946
@ameraner
remove unused sift-devel-complete.pkgs
f09e896
@ameraner
remove deprecated create_distribution.sh
daa1f1d
@ameraner
remove deprecated mtgsift-dist-template, including fill_dir codes. se…
5721feb 
…tting updates codes are not needed anymore with new default configs mechanism
@ameraner
remove deprecated sift.spec, and update build_release.py to use updat…
96f6fab 
…ed sift-pyinstaller-package.spec
@ameraner
remove deprecated sift.spec file
a393ef7
@ameraner
use reduce_data=False when resampling with native, as discussed in py…
32d57d6 
…troll/satpy#2476
@ameraner
Merge branch 'remove_aux_eum_code' into 'master'
d0476c6 
Remove auxiliary and deprecated EUM code

See merge request webservices/mtg-sift!162
@ameraner
Merge branch 'deactivate_reduce_data_for_native_resampling' into 'mas…
787a083 
…ter'

Use reduce_data=False when resampling with native

See merge request webservices/mtg-sift!163
@djhoese
Resurrect globbing creator documentation as a docstring
68173bd
@djhoese
Remove unused boilerplate header of uwswift/control/__init__.py
24ee220
@djhoese
Change CI environments to 3.10 instead of 3.11
4edfc24 
Numba is not currently available on 3.11.
@ameraner
update default values in configs
49a4b00
@ameraner
reorganise default config sorting and file names. create new open_fil…
a08bd2e 
…e_wizard.yaml
@ameraner
move exclude_datasets from data_reading to open_file_wizard, reinstat…
cf2fb30 
…e default_reader.yaml
@ameraner
add couple more readers
334c1bb
@djhoese
Merge branch 'update_default_configs' into 'master'
6314ad8 
Update Default Configs

See merge request webservices/mtg-sift!164
@djhoese
Merge branch 'bugfix-flake8-sphinx' into 'master'
18612ab 
Fix flake8 and sphinx build issues

See merge request webservices/mtg-sift!161
@djhoese
Fix small flake8 issue
3b1de1b
@djhoese
Merge branch 'bugfix-flake8-3' into 'master'
487887b 
Fix small flake8 issue

See merge request webservices/mtg-sift!165
@djhoese
Reformat and fix YAML and python code
08ad953 
This makes pre-commit.ci happy
@djhoese
Merge branch 'style-reformat' into 'master'
bde53eb 
Reformat and fix YAML and python code

See merge request webservices/mtg-sift!166
@djhoese
Move etc/ to package directory
64e16ed
@djhoese
Merge branch 'refactor-etc-pkgdata' into 'master'
7c44657 
Move etc/ to package directory

See merge request webservices/mtg-sift!167
@djhoese djhoese merged commit 44d3323 into master May 14, 2023
@djhoese djhoese deleted the eumetsat_master branch May 14, 2023 13:35
@djhoese djhoese mentioned this pull request May 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@djhoese djhoese

Labels
backwards-incompatibility build: binary installer build: CI Continuous Integration Changes build: conda installation documentation enhancement New feature or request priority: high refactor Housecleaning
Projects
None yet
Milestone
Version 2.0
Development

Successfully merging this pull request may close these issues.
3 participants
@djhoese @askalex @ameraner