Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor CLI entry point #2157

Merged
merged 14 commits into from Apr 21, 2019

Conversation

3 participants
@bartlomieju
Copy link
Contributor

commented Apr 20, 2019

This PR refactors main.rs to use dedicated functions for different deno <subcommand>, as discussed in #2084

EDIT: This PR should be blocked by #2113

EDIT2: As suggested by @kevinkassimo in #2102 and changed --types and --prefetch to respective subcommands:

deno --types -> deno types
deno --prefetch -> deno prefetch

bartlomieju added some commits Apr 20, 2019

@bartlomieju bartlomieju force-pushed the bartlomieju:refactor_cli_entry_point branch from 6e2f8d9 to 172d927 Apr 20, 2019

}).map_err(|(err, _worker)| print_err_and_exit(err))
});
tokio_util::run(main_future);
}

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju Apr 20, 2019

Author Contributor

@ry PTAL at it. It seem that execute_mod_async does not evaluate and run file if prefetch is set to true, so simple letting worker finish the job seems to work fine. I'm not sure though

@bartlomieju bartlomieju force-pushed the bartlomieju:refactor_cli_entry_point branch 2 times, most recently from c51aed8 to 691aea7 Apr 20, 2019

@bartlomieju bartlomieju changed the title [WIP] Refactor CLI entry point Refactor CLI entry point Apr 20, 2019

@bartlomieju bartlomieju force-pushed the bartlomieju:refactor_cli_entry_point branch from 691aea7 to 9d8c7d3 Apr 20, 2019

Show resolved Hide resolved cli/flags.rs Outdated
Show resolved Hide resolved cli/flags.rs
Show resolved Hide resolved cli/flags.rs Outdated
Show resolved Hide resolved cli/main.rs Outdated

@bartlomieju bartlomieju force-pushed the bartlomieju:refactor_cli_entry_point branch from 51c43e5 to 80ef15d Apr 21, 2019

}
);
}
fn flags_from_vec(args: Vec<String>) -> DenoFlags {

This comment has been minimized.

Copy link
@ry

ry Apr 21, 2019

Collaborator

I feel like this function should be the high level public “parse_flags” while the other function which is currently called “parse_flags” should be either combined into this or called something else (and be internal to this module)

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju Apr 21, 2019

Author Contributor

Yeah, I had similar feeling about it, but that means we no longer have matches var available in main(), that gives nice switch on matches.subcommand().

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju Apr 21, 2019

Author Contributor

Alternatively parse_flags may return DenoFlags, ArgMatches to handle both cases

Show resolved Hide resolved cli/flags.rs
Show resolved Hide resolved cli/main.rs Outdated
Show resolved Hide resolved cli/main.rs Outdated
@J2P

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

The manual.md file also seems to need to be modified.

bartlomieju and others added some commits Apr 21, 2019

@ry

ry approved these changes Apr 21, 2019

Copy link
Collaborator

left a comment

LGTM

very nice clean up - thanks !

@ry

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

PS we should add a "deno help" command - it feels like that would be expected now.

@bartlomieju

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2019

@ry no problem, I can add it in an hour or so. I understand you want to add deno help as an alias for deno --help, right?

@ry ry merged commit cd19da6 into denoland:master Apr 21, 2019

3 checks passed

Travis CI - Pull Request Build Passed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@bartlomieju bartlomieju deleted the bartlomieju:refactor_cli_entry_point branch Apr 21, 2019

@justjavac justjavac referenced this pull request Apr 28, 2019

Merged

Upgrade deno v0.3.9 check #28

juniarta added a commit to juniarta/deno that referenced this pull request May 8, 2019

Update (#2)
* Add --no-binary-download option to setup.py (denoland#1916)

* core: Behavior shouldn't be generic

We always pass around Box<[u8]>, and adding this generic is an
unnecessary complication.

Add deno_core_http_bench_test to test.py

sharedQueue works on deno_core_http_bench

* disable arm64 test

* docs: fix example (denoland#1936)

* jsdoc style guide update (denoland#1932)

* Fix TextDecoder for SharedArrayBuffer backed TypedArray (denoland#1940)

* core: remove reset() from SharedQueue API

* Add test style guide (denoland#1918)

* Fix a missing import in manual.md example for testing (denoland#1935)

* Typo fix in styleguide (denoland#1946)

* Fix entry point for tools/docs.py

* Remove unused tools/build_test.py file

* More permissions prompt options (denoland#1926)

* CI jobs should fail quicker on bad test_format or lint (denoland#1954)

* Integrate //core into existing code base

This disables a few tests which are broken still:
- tests/error_004_missing_module.test
- tests/error_005_missing_dynamic_import.test
- tests/error_006_import_ext_failure.test
- repl_test test_set_timeout
- repl_test test_async_op
- repl_test test_set_timeout_interlaced
- all of permission_prompt_test

* Re-order tests for sanity

* Re-enable permissions prompt tests (denoland#1957)

* Remove Eager read, write, accept (denoland#1959)

Removing this reduces tail latency in test/http_bench.ts by half.

* Re-implement init scripts in core (denoland#1958)

Re-enables arm64 CI test

* Deprecate DENO_BUILD_MODE=release

Instead use: tools/build.py --release

* Make write_gn_args.py quiet

* Rename //src/ to //cli/ (denoland#1962)

To better distinguish the deno_core crate from the executable deno,
which will now be called "the cli" internally.

* Add async version of http_util::fetch_sync_string

* Move fetch_remote_source, fetch_local_source

They do not depend on DenoDir, so should not be methods of it.

* Add fetch_remote_source_async (denoland#1972)

* Remove old Buf definition in cli.rs (denoland#1971)

* Separate behavior for the compiler isolate (denoland#1973)

* Add more async module loading function (denoland#1974)

* get_source_code_async
* fetch_module_meta_data_async
* fetch_module_meta_data_and_maybe_compile_async

* Improve pretty printing of objects

If an object has more than 5 elements, it is printed in abbeviated form
displaying only the keys. This is useful in the REPL when inspecting
large objects like the Deno namespace:

  > Deno
  { args, noColor, pid, env, exit, isTTY, execPath, chdir, cwd, File,
  open, stdin, stdout, stderr, read, write, seek, close, copy,
  toAsyncIterator, SeekMode, Buffer, readAll, mkdirSync, mkdir,
  makeTempDirSync, makeTempDir, chmodSync, chmod, removeSync, remove,
  renameSync, rename, readFileSync, readFile, readDirSync, readDir,
  copyFileSync, copyFile, readlinkSync, readlink, statSync, lstatSync,
  stat, lstat, symlinkSync, symlink, writeFileSync, writeFile, ErrorKind,
  DenoError, libdeno, permissions, revokePermission, truncateSync,
  truncate, connect, dial, listen, metrics, resources, run, Process,
  inspect, build, platform, version, Console, stringifyArgs,
  DomIterableMixin }

* pretty-print long strings

* Website and manual improvements (denoland#1967)

* v0.3.4

* Disable arm test again

getting this failure on travis:
https://travis-ci.com/denoland/deno/jobs/186518015

  $ $CARGO_TARGET_DIR/aarch64-unknown-linux-gnu/release/deno tests/002_hello.ts
  qemu: Unsupported syscall: 278
  qemu: Unsupported syscall: 278
  Compiling file:///home/travis/build/denoland/deno/tests/002_hello.ts

* Fix clippy errors and upgrade Rust to 1.33.0 in CI (denoland#1945)

* core: Allow terminating an Isolate from another thread (denoland#1982)

* Remove dead code (denoland#1983)

* Website updates (denoland#1984)

* remove `console` constructor (denoland#1985)

* Handle overflown shared queue (denoland#1992)

Fixes denoland#1988

* Add benchmark for max latency (denoland#1975)

* Update ts_library_builder (denoland#1920)

* Resolve callback moved from Behavior to mod_instantiate()  (denoland#1999)

This simplifies the Behavior trait and makes it more explicit where the
resolve callback is being made.

Also s/StartupScript/Script

* Check `Event` constructor params (denoland#1997)

* namespace reorg: libdeno and DenoCore to Deno.core (denoland#1998)

* fixed typo (denoland#2001)

* Do not close file on invalid seek mode (denoland#2004)

* Fix some typos (denoland#2005)

* remove `isTrusted`'s setter (denoland#2006)

* Fix error when JS is executed with --recompile (denoland#2007)

test case from @afinch7

* Rewrite readFile and writeFile (denoland#2000)

Using open/read/write

* Fix typo (denoland#2010)

* Check `CustomEvent` constructor params (denoland#2011)

* Clippy fixes (denoland#2009)

* Make sharedQueue overflow warning quieter (denoland#2008)

* Merge the --recompile and --reload flags (denoland#2003)

* Add Process.stderrOutput() (denoland#1828)

* v0.3.5

* third_party: upgrade rust crates

* Avoid search prototype chain (denoland#2019)

* check `EventTarget` params (denoland#2018)

* Move //libdeno to //core/libdeno (denoland#2015)

Fixes some sed errors introduced in c43cfe.

Unfortunately moving libdeno required splitting build.rs into two parts,
one for cli and one for core.

I've also removed the arm64 build - it's complicating things at this
re-org and we're not even testing it. I need to swing back to it and get
tools/test.py running for it.

* Call ninja directly from build.rs (denoland#2020)

* Rename crate deno_core to deno (denoland#2022)

* Remove deno_core_http_bench from core/Cargo.toml (denoland#2023)

So we don't have to have an optional tokio dependency. We build
deno_core_http_bench using GN anyway.

* Publish rust crate on crates.io (denoland#2024)

* third_party: add rust crate 'clap'

* Update `deno -h` output in manual (denoland#2030)

* Upgrade to TypeScript 3.4.1 (denoland#2027)

* Improve new TextDecoder().toString() (denoland#2032)

* Add web worker JS API (denoland#1993)

* Refactored the way worker polling is scheduled and errors are handled.
* Share the worker future as a Shared

* add scoop in the install (denoland#2035)

* Follow redirect location as new referrers for nested module imports (denoland#2031)

Fixes denoland#1742
Fixes denoland#2021

* Fix missing return type (denoland#2038)

* Clippy fixes

* cli: refactor compile_sync() for readability

* Add deno_core::RecursiveLoad for async module loading (denoland#2034)

* Refactor deno_core::RecursiveLoad to be more idiomatic (denoland#2034)

* web-compatibility console (denoland#2042)

* fix console.log when error has been caught (denoland#2041)

* fix toString for some web objects (denoland#2040)

* fix JSDoc (denoland#2043)

* Non-fatal compile_sync failures (denoland#2039)

And model worker resources as Stream

* fix code bug in jsdocs (denoland#2048)

* Improve docs in core (denoland#2049)

* v0.3.6

* add compile_async (denoland#2053)

* fix v0.3.6 date error (denoland#2051)

* Add worker benchmarks (denoland#2059)

* Fix tools/docs.py (denoland#2055)

* Test TCP_NODELAY in hyper_hello (denoland#2060)

* Refactor CLI flag parsing (denoland#2025)

* Fix typo in manual.md (denoland#2066)

* Fix unit_tests.py (denoland#2065)

They were silently broken in 780e72 due to flag reordering. This commit
also includes a new assert that would avoid that kind of failure in the
future.

* Flags clean up: DenoFlags::from (denoland#2068)

* use flatbuffer create functions to add fields (denoland#2046)

* Use -O3 instead of -O (denoland#2070)

As recommended in
https://www.reddit.com/r/rust/comments/balb45/why_is_hypers_max_latency_so_high_in_deno/ekck1ww/

* Add link/linkSync fs call for hardlinks (denoland#2074)

* fix `console instanceof Console` (denoland#2073)

* core: snapshot improvements (denoland#2052)

* Moves how snapshots are supplied to the Isolate. Previously they were
  given by Behavior::startup_data() but it was only called once at
  startup. It makes more sense (and simplifies Behavior) to pass it to the
  constructor of Isolate.
* Adds new libdeno type deno_snapshot instead of overloading
  deno_buf.
* Adds new libdeno method to delete snapshot deno_snapshot_delete().
* Renames deno_get_snapshot() to deno_snapshot_new().
* Makes StartupData hold references to snapshots. This was implicit when
  it previously held a deno_buf but is made explicit now. Note that
  include_bytes!() returns a &'static [u8] and we want to avoid
  copying that.

* Allow high precision performance.now() (denoland#1977)

* Merge Worker and Isolate types (denoland#2078)

Reduces generics.

* core: poll ops round robin

Also use a VecDeque to store pending ops to avoid exponential time complexity
when removing completed ops from the list.

* core: Rename Behavior to Dispatch (denoland#2082)

And rename IsolateState to ThreadSafeState.

Also make ThreadSafeState directly implement Dispatch. This is simpler.

* upgrade deno_std & add workaround prettier issue (denoland#2087)

* Silence clippy warnings and format source code

* Use correct type for `source_bytes` when buiding in check-only mode

* fix --v8-options (denoland#2093)

* Improve op dispatch (denoland#2088)

* v0.3.7

* Update `deno -h` output in manual (denoland#2099)

* fix: re-expose DomFile (denoland#2100)

* better wait for removed_from_resource_table_on_close test (denoland#2104)

* Add deno eval subcommand (denoland#2102)

* fix absolute path resolution from remote (denoland#2109)

* Upgrade deno_std (denoland#2111)

* docs: add a pixelated animation logo (denoland#2110)

* Fix silent error, add custom panic handler (denoland#2098)

This is to work around Tokio's panic recovery feature.
Ref tokio-rs/tokio#495
Ref tokio-rs/tokio#209
Ref denoland#1311
Fixes denoland#2097

* third_party: patch hyper to reduce worst-case response latency

* third_party: fix bug in hyper latency patch

* Small fixes in manual.md (denoland#2118)

* third_party: upgrade rust crates

* Add max memory benchmark (denoland#2061)

* Implement async module loading in CLI (denoland#2084)

* core/http_bench: support -D flag to enable logging

* core: run isolate tests within a task

This change is made in preparation for using FuturesUnordered to track
futures that are spawned by the isolate. FuturesUnordered sets up
notififications for every future that it finds to be not ready when
polled, which causes a crash if attempted outside of a task context.

* core: make Isolate use FuturesUnordered to track ops

Additionally, instead of polling ops in a loop until none of them are
ready, the isolate will now yield to the task system after delivering
the first batch of completed ops to the javascript side.

Although this makes performance a bit worse (about 15% fewer
requests/second on the 'deno_core_http_bench' benchmark), we feel that
the advantages are worth it:

* It resolves the extremely high worst-case latency that we were seeing
  on deno_core_http_bench, in particular when using the multi-threaded
  Tokio runtime, which would sometimes exceed a full second.

* Before this patch, the implementation of Isolate::poll() had to loop
  through all sub-futures and poll each one of them, which doesn't scale
  well as the number of futures managed by the isolate goes up. This
  could lead to poor performance when e.g. a server is servicing
  thousands of connected clients.

* Move deno_core_http_bench into examples dir (denoland#2127)

* Add cli dependency on tokio-rustls

* Fix clippy warnings

* benchmarks: fix max_memory and clean up (denoland#2137)

* Fix flaky tests (denoland#2139)

* benchmarks: improve syscall and thread count (denoland#2140)

* Fix redirects under async load (denoland#2133)

* chore: update eslint plugin (denoland#2146)

* avoid prototype builtin hasOwnProperty (denoland#2144)

* Improve test slow_never_ready_modules (denoland#2145)

* core: test Modules::deps and handle error cases better (denoland#2141)

* v0.3.8

* Make Deno/Deno.core not deletable/writable (denoland#2153)

* Exclude .png files from git auto-EOL (denoland#2161)

Fix denoland#1995 for users with the old git version

* fix typo in permissions explanation (denoland#2162)

* docs: improve clarity of the text (denoland#2165)

* Fix link to http_bench example in core README (denoland#2167)

* Refactor CLI entry point (denoland#2157)

Changes "deno --types" to "deno types"
and "deno --prefetch" to "deno prefetch"

* Fixes denoland#2033, shared queue push bug (denoland#2158)

* Fix flaky tests (denoland#2164)

* Remove dead code js/compiler_test.ts (denoland#2160)

* Issue/2170 (denoland#2175)

* Consistency using requiredArguments method

Replaced tuple length check in Headers class with requiredArguments
method.

* Consistency using requiredArguments method

Replaced tuple length check in UrlSearchParams class with
requiredArguments method.

* fmt

* Fix eslint warnings (denoland#2151)

Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: LE GOFF Vincent <g_n_s@hotmail.fr>

* Add Deno.kill(pid, signo) and process.kill(signo) (Unix only) (denoland#2177)

* Simplify logic in URLSearchParams, Buffer, Console (denoland#2174)

* symlink: Ignore type parameter on non-Windows platforms (denoland#2185)

Fixes denoland#2169

* third_party: upgrade rust crates

* Use tokio_threadpool's new panic_handler (denoland#2188)

* core: make Isolate concrete, remove Dispatch trait (denoland#2183)

Op dispatch is now dynamically dispatched, so slightly less efficient.
The immeasurable perf hit is a reasonable trade for the API simplicity
that is gained here.

* Upgrade CI to Node v12 (denoland#2193)

* Fix symlinkSyncNotImplemented (denoland#2198)

* core: Add test for snapshotting from Rust (denoland#2197)

* `URLSearchParams` deleting appended multiple (denoland#2201)

* v0.3.9

* Fix anchor links and add spinner to the benchmarks page (denoland#2205)

* Rename deno prefetch to deno fetch (denoland#2210)

* compile lib.deno_runtime.d.ts into executable (denoland#2209)

* Rename CLI flags to subcommands (denoland#2212)

* v0.3.10

* core: add Deps::to_json() (denoland#2223)

* URLSearchParams init with itself (denoland#2218)

* Use alt logos on benchmarks and manual page (denoland#2211)

* disable --version flag from deno version (denoland#2226)

* recover:  denoland#1517 Kill all pending accepts when TCP listener is closed (denoland#2224)

* fix test description for `atob` (denoland#2232)

* Revert "recover:  denoland#1517 Kill all pending accepts when TCP listener is closed (denoland#2224)" (denoland#2239)

Crashes while running wrk against
js/deps/https/deno.land/std/http/http_bench.ts

This reverts commit 972ac03.

* Disable flaky assert in killSuccess (denoland#2230)

* Add support for custom tsconfig.json (denoland#2089)

Use `--config`

* Rewrite flags.rs::parse_flags (denoland#2237)

* removes ? from URL when deleting all params (denoland#2217)

* flags: fix merge error (denoland#2244)

* Upgrade V8 to 7.6.53

* Fix max_latency benchmark on zero value (denoland#2247)

* Fix another flaky assert (denoland#2246)

* Make `atob` follow the spec (denoland#2242)

* Make error in ast_util more informative. (denoland#2252)

* fs: add Deno.utime/Deno.utimeSync (denoland#2241)

* Rename test targets (denoland#2262)

* core: remove unused function StrBufNullAllocPtr()

* Refactor zero-copy buffers for performance and to prevent memory leaks

* In order to prevent ArrayBuffers from getting garbage collected by V8,
  we used to store a v8::Persistent<ArrayBuffer> in a map. This patch
  introduces a custom ArrayBuffer allocator which doesn't use Persistent
  handles, but instead stores a pointer to the actual ArrayBuffer data
  alongside with a reference count. Since creating Persistent handles
  has quite a bit of overhead, this change significantly increases
  performance. Various HTTP server benchmarks report about 5-10% more
  requests per second than before.

* Previously the Persistent handle that prevented garbage collection had
  to be released manually, and this wasn't always done, which was
  causing memory leaks. This has been resolved by introducing a new
  `PinnedBuf` type in both Rust and C++ that automatically re-enables
  garbage collection when it goes out of scope.

* Zero-copy buffers are now correctly wrapped in an Option if there is a
  possibility that they're not present. This clears up a correctness
  issue where we were creating zero-length slices from a null pointer,
  which is against the rules.

* Async iterator for listener (denoland#2263)

* core: express op as enum (denoland#2255)

* doc: add long about messages for subcommands (denoland#2264)

Type deno <subcommand> -h to view descriptions.

* Add Request global constructor (denoland#2253)

* core: remove support for moving deno_buf ownership from C++ to JavaScript

The functionality hasn't been in use for a long time. Without this feature,
the `alloc_ptr` and `alloc_len` fields are no longer necessary.

* Work around Windows-only V8 concurrent initialization crash

This patch provides a work-around for an apparent V8 bug where
initializing multiple isolates concurrently leads to a crash on
Windows.

At the time of writing the cause of this crash is not exactly
understood, but it seems to be related to the V8 internal
function win64_unwindinfo::RegisterNonABICompliantCodeRange(),
which didn't exist in older versions of V8.

* Re-enable networking unit tests (denoland#2268)

The following tests were commented out in order to get this to go green :
- bodyMultipartFormData
- bodyURLEncodedFormData
- fetchRequestInitStringBody
- netConcurrentAccept
- netListenAsyncIterator

* core,cli: fix clippy warnings

* ci: run clippy

* Optimize read and write ops (denoland#2259)

* fixes body formData tests disabled in denoland#2268 (denoland#2274)

* Fix: deno --v8-options does not print v8 options (denoland#2277)

* add --no-fetch CLI flag to prevent remote downloads (denoland#2213)

* Bump required rustc version to 1.34.1

* ci: make the sanitize build fast again (denoland#2280)

sccache doesn't work for cache debug builds at the moment, because it
doesn't support the `-Xclang -fdebug-compilation-dir` flag that has been
added by the most recent V8 upgrade.

This patch should make the asan/lsan job on Travis CI fast again.

* fix: display "-0" for -0 (denoland#2281)

Added special handling code in js/console.ts

* Minor doc edits (denoland#2231)

* feat: support .mjs extension resolution (denoland#2283)

Removed `extmap` and added .mjs entry in `map_file_extension`.
The assert in the compiler does not need to be updated, since it is
resolving from the compiled cache instead of elsewhere (notice the .map
is asserted next to it)

* feat(cli cmd): deno xeval (denoland#2260)

* add "deno run" subcommand (denoland#2215)

* v0.3.11

* fix: deno version panic (denoland#2286)

* v0.4.0

* add warning for adblockers (denoland#2250)

* update manual.md for deno v0.4.0 (denoland#2291)

* fix: Add all permissions to deno xeval (denoland#2290)

* Add Deno.chown (denoland#2292)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.