Difficulty building dartdevc outside of SDK.

[jonahwilliams]

Current Dart SDK revision is: 71bee8f

When attempting to build either the full dart sdk or when adding a dep on dartdevc to the flutter engine BUILD.gn, the build fails on the //third_party/dart/utils/dartdevc:dartdevc(host) rule. It cannot find the bazel_worker package, however this package is correctly located in our checkout. Interestingly changing the bazel_worker entry in third_party/dart/.packages to a bogus location causes a separate missing file error.

[38/42] ACTION //third_party/dart/utils/dartdevc:dartdevc(//build/toolchain/mac:clang_x64)
FAILED: gen/dartdevc.dart.snapshot 
python ../../third_party/dart/build/gn_run_binary.py compiled_action dart --deterministic --packages=/Users/jonahwilliams/Documents/engine/src/third_party/dart/.packages --snapshot=gen/dartdevc.dart.snapshot --snapshot-depfile=/Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/dartdevc.dart.snapshot.d --snapshot-kind=app-jit /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/bin/dartdevc.dart --dart-sdk /Users/jonahwilliams/Documents/engine/src/third_party/dart/sdk --dart-sdk-summary /Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/third_party/dart/utils/dartdevc/ddc_sdk.sum --library-root /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler -o dartdevc.js /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/bin/dartdevc.dart
Command failed: ./dart --deterministic --packages=/Users/jonahwilliams/Documents/engine/src/third_party/dart/.packages --snapshot=gen/dartdevc.dart.snapshot --snapshot-depfile=/Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/dartdevc.dart.snapshot.d --snapshot-kind=app-jit /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/bin/dartdevc.dart --dart-sdk /Users/jonahwilliams/Documents/engine/src/third_party/dart/sdk --dart-sdk-summary /Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/third_party/dart/utils/dartdevc/ddc_sdk.sum --library-root /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler -o dartdevc.js /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/bin/dartdevc.dart
output: [error] Target of URI doesn't exist: 'package:bazel_worker/bazel_worker.dart'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 13, col 8)
[error] Target of URI doesn't exist: 'package:dev_compiler/src/compiler/shared_command.dart'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 14, col 8)
[error] Undefined name 'ParsedArguments'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 22, col 20)
[error] Undefined class 'StdAsyncWorkerConnection'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 26, col 15)
[error] Undefined class 'SendPortAsyncWorkerConnection'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 27, col 15)
[error] The method 'run' isn't defined for the class '_CompilerWorker'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 28, col 57)
[error] The function 'compile' isn't defined. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 32, col 24)
[error] Classes can only extend other classes. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 38, col 31)
[error] Undefined class 'AsyncWorkerLoop'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 38, col 31)
[error] Undefined class 'ParsedArguments'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 40, col 9)
[error] Undefined class 'AsyncWorkerConnection'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 42, col 38)
[error] The named parameter 'connection' isn't defined. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 43, col 15)
[error] The name 'WorkResponse' isn't a type so it can't be used as a type argument. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 46, col 10)
[error] Undefined class 'WorkRequest'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 46, col 39)
[error] The method 'compile' isn't defined for the class '_CompilerWorker'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 49, col 39)
[error] The method 'WorkResponse' isn't defined for the class '_CompilerWorker'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 53, col 12)
[error] Undefined class 'ParsedArguments'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 60, col 17)
[error] Undefined class 'CompilerResult'. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 68, col 3)
[error] The function 'compile' isn't defined. (../../third_party/dart/pkg/dev_compiler/bin/dartdevc.dart, line 76, col 22)

Please fix all errors before compiling (warnings are okay).

ninja: build stopped: subcommand failed.

To reproduce: checkout flutter/engine according to the instructions at https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment

Under the //third_party/dart:create_sdk dep in https://github.com/flutter/engine/blob/master/BUILD.gn#L25, add a dependency on //third_party/dart:dartdevc`:

You can then either run the command from the error message or build the engine host itself.

./flutter/tools/gn --unopt
ninja -C out/host_debug_unopt

A similar error message is thrown if we attempt to set the dart_platform_sdk gn arg to false. This can be done by updating flutter/tools/gn:

@@ -227,6 +227,7 @@ def to_gn_args(args):
     if args.arm_float_abi:
       gn_args['arm_float_abi'] = args.arm_float_abi
 
+    gn_args['dart_platform_sdk'] = false
     return gn_args
 
 def parse_args(args):

cc @vsmenon

[jonahwilliams]

Actually, there is a second issue when we disable dart_platform_sdk. It appears that the rule is looking for libraries.json via --libraries /Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json, where it should probably be looking under third_party/dart

54/94] ACTION //third_party/dart/utils/dartdevc:dartdevc_kernel_sdk(//build/toolchain/mac:clang_x64)
FAILED: gen/third_party/dart/utils/dartdevc/kernel/ddc_sdk.dill gen/third_party/dart/utils/dartdevc/libraries.json gen/third_party/dart/utils/dartdevc/kernel/amd/dart_sdk.js gen/third_party/dart/utils/dartdevc/kernel/amd/dart_sdk.js.map gen/third_party/dart/utils/dartdevc/kernel/common/dart_sdk.js gen/third_party/dart/utils/dartdevc/kernel/common/dart_sdk.js.map gen/third_party/dart/utils/dartdevc/kernel/es6/dart_sdk.js gen/third_party/dart/utils/dartdevc/kernel/es6/dart_sdk.js.map gen/third_party/dart/utils/dartdevc/kernel/legacy/dart_sdk.js gen/third_party/dart/utils/dartdevc/kernel/legacy/dart_sdk.js.map 
python ../../third_party/dart/build/gn_run_binary.py compiled_action /Users/jonahwilliams/Documents/engine/src/third_party/dart/tools/sdks/dart-sdk/bin/dart --dfe=/Users/jonahwilliams/Documents/engine/src/third_party/dart/tools/sdks/dart-sdk/bin/snapshots/kernel-service.dart.snapshot /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/tool/kernel_sdk.dart --output /Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/third_party/dart/utils/dartdevc/kernel/ddc_sdk.dill --libraries /Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json
Command failed: /Users/jonahwilliams/Documents/engine/src/third_party/dart/tools/sdks/dart-sdk/bin/dart --dfe=/Users/jonahwilliams/Documents/engine/src/third_party/dart/tools/sdks/dart-sdk/bin/snapshots/kernel-service.dart.snapshot /Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/tool/kernel_sdk.dart --output /Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/third_party/dart/utils/dartdevc/kernel/ddc_sdk.dill --libraries /Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json
output: Error: SDK libraries specification not found: file:///Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json.
Normally, the specification is a file named 'libraries.json' in the Dart SDK install location.
Error: Can't infer a .packages file when compiling multiple inputs.
Try specifying the file explicitly with the --packages option.
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_runtime'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_debugger'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_debugger'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_foreign_helper'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_foreign_helper'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_interceptors'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_interceptors'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_internal'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_internal'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_isolate_helper'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_isolate_helper'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_js_helper'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_js_helper'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_js_mirrors'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_js_mirrors'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_js_primitives'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_js_primitives'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_metadata'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_metadata'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:_native_typed_data'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:_native_typed_data'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:async'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:async'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:collection'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:collection'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:convert'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:convert'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:developer'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:developer'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:io'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:io'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:isolate'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:isolate'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:js'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:js'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:js_util'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:js_util'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:math'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:math'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:mirrors'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:mirrors'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:typed_data'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:typed_data'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:indexed_db'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:indexed_db'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:html'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:html'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:html_common'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:html_common'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:svg'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:svg'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:web_audio'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:web_audio'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:web_gl'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:web_gl'
org-dartlang-untranslatable-uri:dart%3A_runtime:1: Error: Not found: 'dart:web_sql'
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: Not found: 'dart:web_sql'
org-dartlang-untranslatable-uri:dart%3A_runtime:1:1: Error: Not found: 'dart:core'
org-dartlang-untranslatable-uri:dart%3Acore:1:8: Error: Not found: 'dart:_internal'
import 'dart:_internal';
       ^
org-dartlang-untranslatable-uri:dart%3Acore:2:8: Error: Not found: 'dart:async'
import 'dart:async';
       ^
org-dartlang-untranslatable-uri:dart%3Acore:4:1: Error: Not found: 'dart:async'
export 'dart:async' show Future, Stream;
^
org-dartlang-untranslatable-uri:dart%3Aasync:23:7: Error: The non-abstract class 'Completer' is missing implementations for these members:
 - Completer.complete
 - Completer.completeError
 - Completer.future
Try to either
 - provide an implementation,
 - inherit an implementation from a superclass or mixin,
 - mark the class as abstract, or
 - provide a 'noSuchMethod' implementation.

class Completer {
      ^^^^^^^^^
org-dartlang-untranslatable-uri:dart%3Aasync:28:3: Context: 'Completer.complete' is defined here.
  complete([value]);
  ^^^^^^^^
org-dartlang-untranslatable-uri:dart%3Aasync:30:3: Context: 'Completer.completeError' is defined here.
  completeError(error, [stackTrace]);
  ^^^^^^^^^^^^^
org-dartlang-untranslatable-uri:dart%3Aasync:26:7: Context: 'Completer.future' is defined here.
  get future;
      ^^^^^^
org-dartlang-untranslatable-uri:dart%3Acore:21:3: Error: The superclass, 'Object', has no unnamed constructor that takes no arguments.
  NoSuchMethodError.withInvocation(receiver, invocation);
  ^^^^^^^^^^^^^^
org-dartlang-untranslatable-uri:dart%3Acore:37:3: Error: The superclass, 'Object', has no unnamed constructor that takes no arguments.
  _InvocationMirror._withType(_memberName, _type, _typeArguments,
  ^^^^^^^^^
org-dartlang-untranslatable-uri:dart%3Aasync:1: Error: The superclass, 'Object', has no unnamed constructor that takes no arguments.
org-dartlang-untranslatable-uri:dart%3Acore:1: Error: The superclass, 'Object', has no unnamed constructor that takes no arguments.
Unhandled exception:
FileSystemException: Cannot copy file to '/Users/jonahwilliams/Documents/engine/src/out/host_debug_unopt/gen/third_party/dart/utils/dartdevc/libraries.json', path = '/Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json' (OS Error: No such file or directory, errno = 2)
#0      _File.throwIfError (dart:io/file_impl.dart:647:7)
#1      _File.copySync (dart:io/file_impl.dart:352:5)
#2      main (file:///Users/jonahwilliams/Documents/engine/src/third_party/dart/pkg/dev_compiler/tool/kernel_sdk.dart:55:25)
<asynchronous suspension>
#3      _startIsolate.<anonymous closure> (dart:isolate/runtime/libisolate_patch.dart:287:32)
#4      _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

[jonahwilliams]

Confirmed that changing
--libraries /Users/jonahwilliams/Documents/engine/src/sdk/lib/libraries.json

to

--libraries /Users/jonahwilliams/Documents/engine/src/third_party/dart/sdk/lib/libraries.json

That particular action passes

[vsmenon]

@jmesserly @jakemac53

Do you know if DDC still has a separate libraries.json?

[jakemac53]

I thought they were merged together already - but I really don't know anything about the sdk builds.

[jonahwilliams]

Might be a bit more to this than just the build rules, since eventually we'll need to include a dart:ui library as well. With dart2js it was as easy as passing a --libraries-spec argument, but It might make sense for to include our own build for dartdevc.

[jonahwilliams]

The second error can be fixed by updating the libraries path in the prebuilt_dart_action("dartdevc_kernel_sdk) rule (at least in the flutter tree, I haven't checked in the SDK repo):

index 039604ae82..3783f8f447 100644
--- a/utils/dartdevc/BUILD.gn
+++ b/utils/dartdevc/BUILD.gn
@@ -347,6 +347,6 @@ prebuilt_dart_action("dartdevc_kernel_sdk") {
     "--output",
     rebase_path(sdk_dill),
     "--libraries",
-    rebase_path("//sdk/lib/libraries.json"),
+    rebase_path("../../sdk/lib/libraries.json"),
   ]
 }

I haven't figured out the issue with the bazel_worker package.

[jonahwilliams]

The error in the dartdevc rule occurs when running the training dart action. Something about the paths there is not configured correctly to run in other repos. I can get it to succeed by invoking the same dart command from //third_party/dart/, or by removing all training args

[vsmenon]

@jonahwilliams If you add a -k to the training args, does that make a difference? That will make dartdevc use the kernel-based front end instead - we're planning to move to that this Q.

[jonahwilliams]

It works!

[vsmenon]

@jonahwilliams - is this working well enough to close? We'll be moving to kernel by default soon.

[jonahwilliams]

Resolved in /dart-lang/sdk/commit/3e304def65677c7b62d43e0a7808fedb9e83a281