Skip to content

SONARPY-957 Protobuf Typeshed should serialize information about variables #1041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrea-guarino-sonarsource merged 3 commits into from
Jan 5, 2022
Merged

SONARPY-957 Protobuf Typeshed should serialize information about variables #1041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b2635c1
SONARPY-957 Serialize information about typeshed variables into protobuf
andreaguarino Jan 3, 2022
511537e
SONARPY-957 Update protobuf files to contain variables
andreaguarino Jan 3, 2022
409a97c
SONARPY-957 Deserialize typeshed variable symbols from protobuf. Fix …
andreaguarino Jan 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 9 additions & 0 deletions its/ruling/src/test/resources/expected/python-S4423.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@
174,
213,
],
'project:tornado-2.3/tornado/simple_httpclient.py':[
210,
],
'project:tornado-2.3/tornado/test/httpserver_test.py':[
110,
115,
120,
125,
],
'project:twisted-12.1.0/doc/core/examples/echoserv_ssl.py':[
15,
],
Expand Down
4 changes: 4 additions & 0 deletions python-checks/src/test/java/org/sonar/python/checks/WeakSSLProtocolCheckTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,8 @@ public void test() {
PythonCheckVerifier.verify("src/test/resources/checks/weakSSLProtocol.py", new WeakSSLProtocolCheck());
}

@Test
public void test_fallback_import() {
PythonCheckVerifier.verify("src/test/resources/checks/weakSSLProtocol_fallback_import.py", new WeakSSLProtocolCheck());
}
}
7 changes: 7 additions & 0 deletions python-checks/src/test/resources/checks/weakSSLProtocol_fallback_import.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
try:
import ssl
except ImportError:
ssl = None

if ssl is not None:
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv2) # Noncompliant
14 changes: 14 additions & 0 deletions python-frontend/src/main/java/org/sonar/python/semantic/SymbolImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand All @@ -39,6 +40,8 @@
import org.sonar.plugins.python.api.types.InferredType;
import org.sonar.python.tree.NameImpl;
import org.sonar.python.types.InferredTypes;
import org.sonar.python.types.TypeShed;
import org.sonar.python.types.protobuf.SymbolsProtos;

public class SymbolImpl implements Symbol {

Expand All @@ -65,6 +68,17 @@ public SymbolImpl(String name, @Nullable String fullyQualifiedName, @Nullable St
this.kind = Kind.OTHER;
}

public SymbolImpl(SymbolsProtos.VarSymbol varSymbol) {
this.name = varSymbol.getName();
this.fullyQualifiedName = TypeShed.normalizedFqn(varSymbol.getFullyQualifiedName());
String fqn = varSymbol.getTypeAnnotation().getFullyQualifiedName();
if (!fqn.isEmpty()) {
Copy link
Copy Markdown
Contributor

@guillaume-dequenne guillaume-dequenne Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to the previous comment, what happens here on a serialized type name that is #Unknown?

Copy link
Copy Markdown
Contributor Author

@andrea-guarino-sonarsource andrea-guarino-sonarsource Jan 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can prevent this to happen in the serializer. We could also add here a check that would throw an exception if type name is #Unknown

Copy link
Copy Markdown
Contributor Author

@andrea-guarino-sonarsource andrea-guarino-sonarsource Jan 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking again about this: #Unknown is the pretty_printed_name which we currently don't really use in the deserializer. The fqn instead should never contain #Unknown.

this.annotatedTypeName = TypeShed.normalizedFqn(fqn);
}
this.validForPythonVersions = new HashSet<>(varSymbol.getValidForList());
this.kind = Kind.OTHER;
}

@Override
public String name() {
return name;
Expand Down
9 changes: 8 additions & 1 deletion python-frontend/src/main/java/org/sonar/python/types/TypeShed.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,13 +403,17 @@ static Map<String, Symbol> getSymbolsFromProtobufModule(@Nullable ModuleSymbol m
moduleSymbol.getOverloadedFunctionsList().stream()
.filter(d -> isValidForProjectPythonVersion(d.getValidForList()))
.forEach(proto -> descriptorsByName.computeIfAbsent(proto.getName(), d -> new HashSet<>()).add(proto));
moduleSymbol.getVarsList().stream()
.filter(d -> isValidForProjectPythonVersion(d.getValidForList()))
.forEach(proto -> descriptorsByName.computeIfAbsent(proto.getName(), d -> new HashSet<>()).add(proto));

Map<String, Symbol> deserializedSymbols = new HashMap<>();

for (Map.Entry<String, Set<Object>> entry : descriptorsByName.entrySet()) {
String name = entry.getKey();
Set<Symbol> symbols = symbolsFromDescriptor(entry.getValue(), false);
deserializedSymbols.put(name, disambiguateSymbolsWithSameName(name, symbols, moduleSymbol.getFullyQualifiedName()));
Symbol disambiguatedSymbol = disambiguateSymbolsWithSameName(name, symbols, moduleSymbol.getFullyQualifiedName());
deserializedSymbols.put(name, disambiguatedSymbol);
}
return deserializedSymbols;
}
Expand Down Expand Up @@ -465,6 +469,9 @@ public static Set<Symbol> symbolsFromDescriptor(Set<Object> descriptors, boolean
}
symbols.add(fromOverloadedFunction(((OverloadedFunctionSymbol) descriptor), isInsideClass));
}
if (descriptor instanceof SymbolsProtos.VarSymbol) {
symbols.add(new SymbolImpl((SymbolsProtos.VarSymbol) descriptor));
}
}
return symbols;
}
Expand Down
8 changes: 8 additions & 0 deletions python-frontend/src/main/protobuf/symbols.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,4 +88,12 @@ message ModuleSymbol {
repeated ClassSymbol classes = 2;
repeated FunctionSymbol functions = 3;
repeated OverloadedFunctionSymbol overloaded_functions = 4;
repeated VarSymbol vars = 5;
}

message VarSymbol {
string name = 1;
string fully_qualified_name = 2;
optional Type type_annotation = 3;
repeated string valid_for = 4;
}
15 changes: 14 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/2@ConfigParser.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,4 +573,17 @@ has_option'ConfigParser.RawConfigParser.has_option"
Any
Any"builtins.dict*'
depth
builtins.int" builtins.intz27j27
builtins.int" builtins.intz27j27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27*I
DEFAULTSECTConfigParser.DEFAULTSECT
builtins.str" builtins.str"27*a
MAX_INTERPOLATION_DEPTH$ConfigParser.MAX_INTERPOLATION_DEPTH
builtins.int" builtins.int"27
Expand Down
9 changes: 8 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/2@Queue.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,11 @@ get_nowaitQueue.Queue.get_nowait"
Queue.Queue[Queue._T]
Queue._T" Queue.Queuez27Pj277
PriorityQueueQueue.PriorityQueue" Queue.QueuePj27/
LifoQueueQueue.LifoQueue" Queue.QueuePj27
LifoQueueQueue.LifoQueue" Queue.QueuePj27*9
Any
typing.Any"
builtins.object"builtins.object"27*I
Generictyping.Generic*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
18 changes: 17 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/2@SocketServer.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,4 +224,20 @@ $Union[builtins.unicode,builtins.str]$
selfB
SocketServer.BaseRequestHandler"SocketServer.BaseRequestHandlerz27j27^
StreamRequestHandler!SocketServer.StreamRequestHandler"SocketServer.BaseRequestHandlerj27b
DatagramRequestHandler#SocketServer.DatagramRequestHandler"SocketServer.BaseRequestHandlerj27
DatagramRequestHandler#SocketServer.DatagramRequestHandler"SocketServer.BaseRequestHandlerj27*,

SocketTypesocket.SocketType
Any"27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Callabletyping.Callable*
typing._SpecialForm"typing._SpecialForm"27*K
ClassVartyping.ClassVar*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
13 changes: 12 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/BaseHTTPServer.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,4 +236,15 @@ send_error0BaseHTTPServer.BaseHTTPRequestHandler.send_error"
address_string4BaseHTTPServer.BaseHTTPRequestHandler.address_string"
builtins.str" builtins.str*X
selfN
%BaseHTTPServer.BaseHTTPRequestHandler"%BaseHTTPServer.BaseHTTPRequestHandlerz27j27
%BaseHTTPServer.BaseHTTPRequestHandler"%BaseHTTPServer.BaseHTTPRequestHandlerz27j27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Callabletyping.Callable*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
9 changes: 8 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/Cookie.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,4 +125,11 @@ BaseCookieCookie.BaseCookie" builtins.dict*E
valz27*F
value_encodeCookie.SmartCookie.value_encode*
self*
valz27j27
valz27j27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*$
CookieCookie.Cookie
Any"27
Expand Down
4 changes: 3 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/HTMLParser.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,4 +139,6 @@ HTMLParserHTMLParser.HTMLParser"markupbase.ParserBase*r
HTMLParser.HTMLParser"HTMLParser.HTMLParser*
s
typing.AnyStrz27j27C
HTMLParseErrorHTMLParser.HTMLParseError"builtins.Exceptionj27
HTMLParseErrorHTMLParser.HTMLParseError"builtins.Exceptionj27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
9 changes: 8 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/SimpleHTTPServer.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,4 +450,11 @@ guess_type4SimpleHTTPServer.SimpleHTTPRequestHandler.guess_type"
pathl
$Union[builtins.str,builtins.unicode]
builtins.str" builtins.str$
builtins.unicode"builtins.unicodez27j27
builtins.unicode"builtins.unicodez27j27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
9 changes: 8 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/StringIO.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,4 +303,11 @@ writelinesStringIO.StringIO.writelines"
builtins.bool"builtins.bool*R
selfH
StringIO.StringIO[typing.AnyStr]
typing.AnyStr"StringIO.StringIOz27Pj27
typing.AnyStr"StringIO.StringIOz27Pj27*9
Any
typing.Any"
builtins.object"builtins.object"27*I
Generictyping.Generic*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
13 changes: 12 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/UserDict.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,4 +335,15 @@ setdefaultUserDict.DictMixin.setdefault"
overloadtyping.overload"
typing._F*
f
typing._Fz27
typing._Fz27*9
Any
typing.Any"
builtins.object"builtins.object"27*I
Generictyping.Generic*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
4 changes: 3 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/UserList.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,4 +196,6 @@
overloadtyping.overload"
typing._F*
f
typing._Fz27
typing._Fz27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
11 changes: 10 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/UserString.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -619,4 +619,13 @@ startswith UserString.UserString.startswith"
overloadtyping.overload"
typing._F*
f
typing._Fz27
typing._Fz27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27
Expand Down
25 changes: 24 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/__builtin__.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6002,4 +6002,27 @@ VTuple[__builtin__._T1,__builtin__._T2,__builtin__._T3,__builtin__._T4,__builtin
Any"typing.Iterable*?
iterables0
typing.Iterable[Any]
Any"typing.Iterable0:overloadX"27
Any"typing.Iterable0:overloadX"27*9
Any
typing.Any"
builtins.object"builtins.object"27*K
Callabletyping.Callable*
typing._SpecialForm"typing._SpecialForm"27*I
Generictyping.Generic*
typing._SpecialForm"typing._SpecialForm"27*K
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27*K
Protocoltyping.Protocol*
typing._SpecialForm"typing._SpecialForm"27*E
Tuple typing.Tuple*
typing._SpecialForm"typing._SpecialForm"27*C
Type typing.Type*
typing._SpecialForm"typing._SpecialForm"27*E
Union typing.Union*
typing._SpecialForm"typing._SpecialForm"27*j
Literaltyping_extensions.Literal@
typing_extensions._SpecialForm"typing_extensions._SpecialForm"27*t
NotImplemented__builtin__.NotImplementedB
__builtin__._NotImplementedType"__builtin__._NotImplementedType"27*R
Ellipsis__builtin__.Ellipsis,
__builtin__.ellipsis"__builtin__.ellipsis"27
Expand Down
29 changes: 28 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/__future__.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,31 @@ GTuple[builtins.int,builtins.int,builtins.int,builtins.str,builtins.int]
builtins.str" builtins.str
builtins.int" builtins.int*4
self*
__future__._Feature"__future__._Featurez27z35z36z37z38z39z310j27j35j36j37j38j39j310
__future__._Feature"__future__._Featurez27z35z36z37z38z39z310j27j35j36j37j38j39j310*v
absolute_import__future__.absolute_import*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*h
division__future__.division*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*l

generators__future__.generators*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*r
nested_scopes__future__.nested_scopes*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*t
print_function__future__.print_function*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*x
unicode_literals__future__.unicode_literals*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*t
with_statement__future__.with_statement*
__future__._Feature"__future__._Feature"27"35"36"37"38"39"310*�
all_feature_names__future__.all_feature_namesJ
builtins.list[builtins.str]
builtins.str" builtins.str"builtins.list"27"35"36"37"38"39"310*p
barry_as_FLUFL__future__.barry_as_FLUFL*
__future__._Feature"__future__._Feature"35"36"37"38"39"310*p
generator_stop__future__.generator_stop*
__future__._Feature"__future__._Feature"35"36"37"38"39"310*�
__all____future__.__all__J
builtins.list[builtins.str]
builtins.str" builtins.str"builtins.list"35"36"37"38"39"310*b
annotations__future__.annotations*
__future__._Feature"__future__._Feature"37"38"39"310
Expand Down
5 changes: 4 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/__main__.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@
__getattr____main__.__getattr__"
Any*$
name
builtins.str" builtins.strz27z35z36z37z38z39z310
builtins.str" builtins.strz27z35z36z37z38z39z310*R
Any
typing.Any"
builtins.object"builtins.object"27"35"36"37"38"39"310
19 changes: 18 additions & 1 deletion python-frontend/src/main/resources/org/sonar/python/types/protobuf/_ast.protobuf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,4 +189,21 @@ MatchValue_ast.MatchValue" _ast.patternj3108

MatchClass_ast.MatchClass" _ast.patternj310*
MatchAs _ast.MatchAs" _ast.patternj310*
MatchOr _ast.MatchOr" _ast.patternj310
MatchOr _ast.MatchOr" _ast.patternj310*d
Optionaltyping.Optional*
typing._SpecialForm"typing._SpecialForm"27"35"36"37"38"39"310*A
__version___ast.__version__
builtins.str" builtins.str"27*^
PyCF_ONLY_AST_ast.PyCF_ONLY_AST
builtins.int" builtins.int"27"35"36"37"38"39"310*N
Any
typing.Any"
builtins.object"builtins.object"35"36"37"38"39"310*`
ClassVartyping.ClassVar*
typing._SpecialForm"typing._SpecialForm"35"36"37"38"39"310*
Literaltyping_extensions.Literal@
typing_extensions._SpecialForm"typing_extensions._SpecialForm"35"36"37"38"39"310*X
PyCF_TYPE_COMMENTS_ast.PyCF_TYPE_COMMENTS
builtins.int" builtins.int"38"39"310*h
PyCF_ALLOW_TOP_LEVEL_AWAIT_ast.PyCF_ALLOW_TOP_LEVEL_AWAIT
builtins.int" builtins.int"38"39"310
Expand Down
Loading