Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dart:ffi analysis rules in dart analyzer #35777

Closed
dcharkes opened this issue Jan 25, 2019 · 15 comments
Closed

dart:ffi analysis rules in dart analyzer #35777

dcharkes opened this issue Jan 25, 2019 · 15 comments
Labels
area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. library-ffi

Comments

@dcharkes
Copy link
Contributor

The Dart compiler rejects programs dart:ffi if the C and Dart types do not match up:

Pointer<Int32> p;
p.store(10.0); // only Dart [int] can be stored into C [Int32]

However, the Dart analyzer does not report these errors yet.

Implement the same static checks in Dart analyzer.

@dcharkes dcharkes added this to the Dart VM FFI 1.0 milestone Jan 25, 2019
@dcharkes dcharkes self-assigned this Jan 25, 2019
@kevmoo kevmoo added the area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. label Jan 25, 2019
@lrhn
Copy link
Member

lrhn commented Jan 28, 2019

Are we absolutely sure we cannot encode these type requirements in the Dart types somehow, rather than introduce a completely new type system that the analyzer needs to know about?

I can see that it is annoying to have to write Pointer<Int32, int>, but we can't deconstruct types in type applications, so even if Int32 was carrying the int, we can't extract it. Maybe if we have type aliases, we can do:

typedef Int32p = Pointer<Int32, int>;
typedef Int16p = Pointer<Int16, int>;
typedef Float64p = Pointer<Float64, double>;

Or, the type Int32 can be defined something like:

class CType<T> {
  Pointer<CType<T>, T> get NULL;
}
class Int32 extends CType<int> {
   Pointer<Int32, int> get NULL => Pointer<Int32, int>.NULL();
}
class Pointer<C extends CType<T>, T> { 
  Pointer.NULL()  : ...;
}

and then you can write:

var p = Int32.NULL;

to initialize a pointer to the full type without having to write it.

@mit-mit mit-mit removed this from the Dart VM FFI 1.0 milestone Mar 11, 2019
@leafpetersen
Copy link
Member

@MichaelRFairhurst @stereotype441 @bwilkerson @scheglov Is this the sort of thing that might best be done in a plugin? What would be involved in that?

@dcharkes dcharkes assigned sjindel-google and unassigned dcharkes Sep 19, 2019
@dcharkes
Copy link
Contributor Author

As discussed last meeting, assigning @sjindel-google.

@MichaelRFairhurst
Copy link
Contributor

Note that we are not entirely sure that the performance of plugins is good enough for plugins in the current design.

Definitely the first option that should be explored is using the dart type checker normally.

@sjindel-google
Copy link
Contributor

Definitely the first option that should be explored is using the dart type checker normally.

Could you please elaborate on this? I was planning to pursue implementing this as hints, given @bwilkerson's feedback on dart-lang/linter#1734.

@MichaelRFairhurst
Copy link
Contributor

MichaelRFairhurst commented Sep 24, 2019

by "using the dart type checker normally" I meant like @lrhn's suggestion.

Mostly my comment was in terms of whether we should explore using an analysis plugin. I do not recommend an analysis plugin as this purpose unless all other avenues have been seriously considered. Looks like lints fall into that category of "all other avenues."

Though lints have problems too...even if they are enabled by default, it is not ideal to have hints in the analyzer for something that is a compile time error in the CFE. But I'm assuming that these trade-offs have already been thought through pretty thoroughly.

@bwilkerson
Copy link
Member

tl;dr Implementing these diagnostics as hints is the best implementation strategy.

I agree that we don't want to use a plugin for these. In addition to the potential performance issues, we want these to always be enabled. Plugins are like lints in that respect because users have to add a dependency to their package in order to allow the plugin to be loaded.

Note that a hint is just a diagnostic that isn't defined by the Dart Language Specification. The severity of the hint can be set to "error", if that's what we want, and users will see it exactly the same was as any of the diagnostics defined in the spec.

@dcharkes
Copy link
Contributor Author

by "using the dart type checker normally" I meant like @lrhn's suggestion.

The simple cases with Int8 and int will be part of the normal type system through extension methods.

extension PointerInt8 on Pointer<Int8> {
  int load();
  store(int v);
}
// etc.

The things which cannot be done in the Dart type system is pairwise checking types in two function signatures for an arbitrary length signature:

Pointer<NativeFunction<Int8 Function(Int8, Int8)>> p;
final f = p.asFunction<int Function(int, int)>();

I thought @lrhn, @sjindel-google, and I had discussed that (half a year ago), but I cannot find the issue/post on GitHub. (Maybe it was on a whiteboard.)

@sjindel-google
Copy link
Contributor

There are also non-type-system related checks, like:

  • No extending core FFI classes
  • No extending structs
  • No generic structs
  • Parameters to fromFunction must be constants,
    etc.

However, we should be using isSubtypeOf for implementing the function checks, both in the CFE and Analyzer, to avoid variance errors.

@dcharkes dcharkes added this to the D26 Release milestone Sep 30, 2019
@mkustermann
Copy link
Member

We're enabling the analyzer to run on our ffi test suite. I'll approve the following failures

BOT TEST RESULT EXPECTED
analyzer-analysis-server-linux ffi/enable_ffi_test/01 Pass CompileTimeError
analyzer-analysis-server-linux ffi/function_callbacks_test/59 Pass CompileTimeError
analyzer-analysis-server-linux ffi/function_callbacks_test/60 Pass CompileTimeError
analyzer-analysis-server-linux ffi/function_callbacks_test/61 Pass CompileTimeError
analyzer-analysis-server-linux ffi/function_callbacks_test/62 Pass CompileTimeError
analyzer-analysis-server-linux ffi/function_callbacks_test/63 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/11 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/12 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/13 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/14 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/15 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/16 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/17 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/18 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/50 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/52 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/53 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/54 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/56 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/57 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/58 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/59 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/60 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/70 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/71 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/72 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/73 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/74 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/75 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/76 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/80 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/81 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/810 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/811 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/812 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/813 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/82 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/83 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/84 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/85 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/86 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/87 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/88 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/90 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/91 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/910 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/911 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/912 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/913 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/92 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/93 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/94 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/95 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/96 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/97 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/98 Pass CompileTimeError
analyzer-analysis-server-linux ffi/static_checks_test/99 Pass CompileTimeError
analyzer-linux-release/asserts ffi/enable_ffi_test/01 Pass CompileTimeError
analyzer-linux-release/asserts ffi/function_callbacks_test/59 Pass CompileTimeError
analyzer-linux-release/asserts ffi/function_callbacks_test/60 Pass CompileTimeError
analyzer-linux-release/asserts ffi/function_callbacks_test/61 Pass CompileTimeError
analyzer-linux-release/asserts ffi/function_callbacks_test/62 Pass CompileTimeError
analyzer-linux-release/asserts ffi/function_callbacks_test/63 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/11 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/12 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/13 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/14 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/15 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/16 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/17 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/18 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/50 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/52 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/53 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/54 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/56 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/57 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/58 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/59 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/60 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/70 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/71 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/72 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/73 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/74 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/75 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/76 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/80 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/81 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/810 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/811 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/812 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/813 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/82 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/83 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/84 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/85 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/86 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/87 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/88 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/90 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/91 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/910 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/911 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/912 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/913 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/92 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/93 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/94 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/95 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/96 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/97 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/98 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/99 Pass CompileTimeError
analyzer-mac-release/asserts ffi/enable_ffi_test/01 Pass CompileTimeError
analyzer-mac-release/asserts ffi/function_callbacks_test/59 Pass CompileTimeError
analyzer-mac-release/asserts ffi/function_callbacks_test/60 Pass CompileTimeError
analyzer-mac-release/asserts ffi/function_callbacks_test/61 Pass CompileTimeError
analyzer-mac-release/asserts ffi/function_callbacks_test/62 Pass CompileTimeError
analyzer-mac-release/asserts ffi/function_callbacks_test/63 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/11 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/12 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/13 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/14 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/15 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/16 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/17 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/18 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/50 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/52 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/53 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/54 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/56 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/57 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/58 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/59 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/60 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/70 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/71 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/72 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/73 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/74 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/75 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/76 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/80 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/81 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/810 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/811 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/812 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/813 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/82 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/83 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/84 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/85 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/86 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/87 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/88 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/90 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/91 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/910 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/911 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/912 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/913 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/92 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/93 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/94 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/95 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/96 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/97 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/98 Pass CompileTimeError
analyzer-mac-release/asserts ffi/static_checks_test/99 Pass CompileTimeError

dart-bot pushed a commit that referenced this issue Oct 10, 2019
Issue #35777

Change-Id: Ia8edd9901d52e671a116c7747319fdadd7da4681
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121060
Reviewed-by: Daco Harkes <dacoharkes@google.com>
Reviewed-by: Samir Jindel <sjindel@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
@mkustermann
Copy link
Member

@bwilkerson Thank you so much for helping us on this - very much appreciate it!

@dcharkes
Copy link
Contributor Author

@mkustermann I've preapproved the following test as well.

BOT TEST RESULT EXPECTED
analyzer-analysis-server-linux ffi/static_checks_test/815 Pass CompileTimeError
analyzer-linux-release/asserts ffi/static_checks_test/815 Pass CompileTimeError

@mkustermann
Copy link
Member

I've uploaded cl/121711 with the remaining changes to the analyzer, which should make the analyzer pass our ffi test suite.

dart-bot pushed a commit that referenced this issue Oct 15, 2019
This makes all tests in the ffi test suite pass:

   % tools/test.py -cdart2analyzer -mrelease -ax64 ffi

Issue #35777

Change-Id: I93338ee530041e5e8cb1eb5958b12fbf1517496e
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/121711
Auto-Submit: Martin Kustermann <kustermann@google.com>
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
@mkustermann
Copy link
Member

The main checks have been implemented. We'll probably want to fine tune the error messages to make CFE and Analyzer provide roughly the same error message.

Though I don't consider this blocking for D26. So removing the milestone label.

@mkustermann mkustermann removed this from the D26 Release milestone Oct 16, 2019
@mkustermann mkustermann removed their assignment Oct 16, 2019
@mkustermann
Copy link
Member

Filed #38970 for the alignment of error messages.

We consider this bug to be fixed - the analyzer has FFI support now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. library-ffi
Projects
None yet
Development

No branches or pull requests

9 participants