Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli cmd): deno xeval #2260

Merged
merged 7 commits into from May 3, 2019

Conversation

5 participants
@kevinkassimo
Copy link
Contributor

commented May 1, 2019

Prototype of #2257.
Quick prototype for deno xeval. Allows an optional -I/--replvar="varname" flag to set varname to access (if not provided, set to $).

I'll polish the code and try adding tests tomorrow

e.g.
For file list.txt:

a
bbbbbb
c  d e     f


gggg
$ cat list.txt | ./target/debug/deno -I="val" "console.log(val);"
a
bbbbbb
c
d
e
f
gggg
$ cat list.txt | ./target/debug/deno xeval -I="val" "window.cache = window.cache || []; window.cache.push(val); console.log(window.cache.join(''));"
a
abbbbbb
abbbbbbc
abbbbbbcd
abbbbbbcde
abbbbbbcdef
abbbbbbcdefgggg
$ cat list.txt | ./target/debug/deno -A xeval "(async () => { console.log('isFile:', (await Deno.stat('.')).isFile()); console.log($); })()"
isFile: false
a
isFile: false
bbbbbb
isFile: false
c
isFile: false
d
isFile: false
e
isFile: false
f
isFile: false
gggg
Show resolved Hide resolved cli/flags.rs Outdated
.arg(
Arg::with_name("var")
.long("var")
.short("V")

This comment has been minimized.

Copy link
@ry

ry May 1, 2019

Collaborator

I'd prefer

          Arg::with_name("replvar")
            .short("I")
@ry

This comment has been minimized.

Copy link
Collaborator

commented May 1, 2019

Can you add an example to the manual?

It would also be good to have an example in "deno help xeval"

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

@ry will do (with other changes) tomorrow

@bartlomieju

This comment has been minimized.

Copy link
Contributor

commented May 1, 2019

@kevinkassimo given your example:
list.txt

a
bbbbbb
c  d e     f


gggg

Shouldn't output look like this?

$ cat list.txt | ./target/debug/deno -V="val" "console.log(val);"
a
bbbbbb
c  d e     f


gggg

In your example it treats whitespace as line breaks, is it intended?

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

@bartlomieju intended, although I'm not so sure if this is the best way to approach... or should we just use newline only?

It seems that behavior of xargs on different platforms also slightly differs. For the sample file,

# macOS
$ cat list.txt | xargs -I% echo '%'
a
bbbbbb
c d e f gggg

# Linux(Ubuntu)
$ cat list.txt | xargs -I% echo %
a
bbbbbb
c  d e     f
gggg

also linux version of xargs also supports -d for delimiter -- which does not exist on macOS.

@bartlomieju bartlomieju referenced this pull request May 1, 2019

Merged

add run CLI subcommand #2215

Arg::with_name("var")
.long("var")
.short("V")
.about("`xargs`-like eval script")

This comment has been minimized.

Copy link
@ry

ry May 1, 2019

Collaborator

No backticks please

Also I don't think we should assume people know what xargs is. It would be clearer to write "eval a script on lines from stdin"

",
).arg(
Arg::with_name("replvar")
.long("replvar")

This comment has been minimized.

Copy link
@ry

ry May 1, 2019

Collaborator

Is long necessary?

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju May 1, 2019

Contributor

It is, otherwise this big message would show up in deno -h. With long_about it will show up in `deno help xeval

Sorry, you meant other Arg.long, then it's not needed.

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 1, 2019

Author Contributor

I think it is necessary if you want to support long flags, e.g. --replvar

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju May 1, 2019

Contributor

@kevinkassimo you're right, sorry for noise

Show resolved Hide resolved cli/flags.rs Outdated
@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

Introduced -d/--delimiter flag for setting custom delimiter. Default to newline.

# Default delimiting with newline
$ echo -ne 'a\nb\n\nc  c' | ./target/debug/deno xeval -I "val" "console.log(val)"
a
b
c  c
# Use space for delimiting
$ echo -ne 'a b   c' | ./target/debug/deno xeval -I "val" -d ' ' "console.log(val)"
a
b
c
# Unicode
$ echo -ne 'a分b分c分分口d' | ./target/debug/deno xeval -I "val" -d '' "console.log(val)"
a
b
c
口d
# Multicharacter delimiter
$ echo -ne '1DELIM2DELIMDELIM3' | ./target/debug/deno xeval -I "val" -d 'DELIM' "console.log(val)"
1
2
3

Currently, clap does not escape input string, so -d "\n" will treat "\n" literally. We either have to use -d $'\n' or to introduce a package that does the escaping work.

# echo also have '\n' interpreted literally without `-e`
$ echo -n 'a\nb' | ./target/debug/deno xeval -I "val" -d '\n' "console.log(val)"
a
b
# we need $'\n' to use newline as delimiter
$ echo -ne 'a\nb' | ./target/debug/deno xeval -I "val" -d $'\n' "console.log(val)"
a
b

If the current design is approved, I will go and add examples to manual and add integration tests.

@kevinkassimo kevinkassimo force-pushed the kevinkassimo:cmd/xeval branch from a530280 to 553836c May 1, 2019

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

Also help output:

$ ./target/debug/deno xeval -h
deno-xeval

Eval a script on lines (or chunks split under delimiter) from stdin.

Read from standard input and eval code on each whitespace-delimited
string chunks.

-I/--replvar optionally set variable name for input to be used in eval.
Otherwise '$' will be used as default variable name.

  cat list.txt | deno xeval 'console.log($)'
  cat list.txt | deno xeval -I 'val' 'console.log(val)'
  cat list.txt | deno xeval -d ' ' 'console.log($)'
  cat list.txt | deno xeval -d $'\n' 'console.log($)'

USAGE:
    deno xeval [OPTIONS] <code>

FLAGS:
    -h, --help    Prints help information

OPTIONS:
    -d <delim>          Set delimiter, defaults to newline
    -I <replvar>        Set variable name to be used in eval, defaults to $

ARGS:
    <code>
@bartlomieju

This comment has been minimized.

Copy link
Contributor

commented May 1, 2019

Currently, clap does not escape input string, so -d "\n" will treat "\n" literally. We either have to use -d $'\n' or to introduce a package that does the escaping work.

Can you elaborate? It looks like both provided examples behave the same.

EDIT: @kevinkassimo this feature looks awesome

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

@bartlomieju For example:

$ echo -ne "a\nb\n\n\n" | ./target/debug/deno xeval "console.log($)"
a
b
$ echo -ne "a\nb\n\n\n" | ./target/debug/deno xeval -d "\n" "console.log($)"
a
b


# Notice the above newlines. Since xeval would ignore empty segments, 
# this means that these newlines are not used as delimiter, but treated as part of each text segment

# $'\n' uses shell feature, so '\n' is unescaped correctly and this works
$ echo -ne "a\nb\n\n\n" | ./target/debug/deno xeval -d $'\n' "console.log($)"
a
b

@kevinkassimo kevinkassimo force-pushed the kevinkassimo:cmd/xeval branch from 553836c to 9dd5bca May 1, 2019

@ry ry force-pushed the kevinkassimo:cmd/xeval branch from 9dd5bca to 773962e May 2, 2019

@ry

This comment has been minimized.

Copy link
Collaborator

commented May 2, 2019

I rebased this PR

@kevinkassimo kevinkassimo force-pushed the kevinkassimo:cmd/xeval branch 2 times, most recently from 6c7c729 to 763c7dc May 3, 2019

@ry

This comment has been minimized.

Copy link
Collaborator

commented May 3, 2019

Failing on windows

tests/030_xeval.test ... FAILED
2683Expected output does not match actual.
2684Expected output:
2685A
2686B
2687C
2688
2689Actual output:
2690A
2691
2692B
2693
2694
2695
2696C
2697
2698
2699

@kevinkassimo kevinkassimo force-pushed the kevinkassimo:cmd/xeval branch 3 times, most recently from 65f1f56 to ca5e183 May 3, 2019

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

@ry resolved. Seems to be some funny problem with \r\n on windows

cat list.txt | deno xeval 'console.log($)'
cat list.txt | deno xeval -I 'val' 'console.log(val)'
cat list.txt | deno xeval -d ' ' 'console.log($)'
cat list.txt | deno xeval -d $'\\n' 'console.log($)'

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

I think examples that people can copy, paste, and run out of the box would be better. In particular, if we use a file that usually exists. Say /etc/passwd.

Here's one that prints all the users on the system:

cat /etc/passwd | ./target/debug/deno xeval "a = $.split(':'); if (a) console.log(a[0])"

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

This isn't xeval, but normal eval. Here's an obfuscated way of counting the bytes of a file:

cat README.md | ./target/debug/deno eval "b = await Deno.readAll(Deno.stdin); console.log(b.length)"

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 3, 2019

Author Contributor

Or something like

git branch | deno xeval "if ($.startsWith('*')) console.log($.split(' ')[1])"

to find current branch?

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

Yes that's good. I'd modify it to be:

git branch | deno xeval "if ($.startsWith('*')) console.log($.slice(2))"

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 3, 2019

Author Contributor

I also try to find an example to use the -d flag. A quite contrived example is to check if somthing is MIT licensed:

cat LICENSE | deno xeval -d " " "if ($ === 'MIT') console.log('MIT licensed')"

If this works, my plan is to include the following 3 examples in deno xeval help:

cat /etc/passwd | deno xeval "a = $.split(':'); if (a) console.log(a[0])"
git branch | deno xeval -I 'line' "if (line.startsWith('*')) console.log(line.slice(2))"
cat LICENSE | deno xeval -d " " "if ($ === 'MIT') console.log('MIT licensed')"

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

SGTM

kevinkassimo added some commits May 1, 2019

feat(cli cmd): deno xeval prototype
Prototype for `deno xeval`. Allows an optional `-V/--var="varname"` flag to set varname to access
fix: delete unintended line
(unexpectedly included code for experiment...)
feat: add flag for delimiter
-d/--delim for delimiter.
Currently, clap does not escape input string, so -d "\n" will treat "\n"
literally. We either have to use -d $'\n' or to introduce a package that
does the escaping work.

kevinkassimo added some commits May 3, 2019

test: add tests
Added 3 integration tests

@kevinkassimo kevinkassimo force-pushed the kevinkassimo:cmd/xeval branch from ca5e183 to d4ef1b1 May 3, 2019

let xeval_delim = state
.flags
.xeval_delim
.clone()

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

Is the clone necessary?

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 3, 2019

Author Contributor

I believe yes, we are doing a move later as we initialize the StartResArgs struct (while state.flags.log_debug copies by default)

const sliceRead = inspectArr.subarray(0, rr.nread);
// Avoid unicode problems
let nextSliceStartIndex = 0;
for (let i = 0; i < sliceRead.length; i++) {

This comment has been minimized.

Copy link
@ry

ry May 3, 2019

Collaborator

Can you abstract out a function called lines() which yields a generator of strings. I believe @bartlomieju might have done a version of this before. Add a TODO above function stating that this functionality exists in deno_std, and that we should dedup the code once we have the ability to import from there.

for await (let line of lines(stdin)) {
  // ...
}

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 3, 2019

Author Contributor

Did not find this utility in deno_std. I'll add a comment nevertheless, saying we might want to port this to deno_std and later import from it once it is doable.

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju May 3, 2019

Contributor

I made a try in #2101 but we didn't land it.

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 3, 2019

Author Contributor

@ry I instead abstracted out chunks(reader, delim) that supports any string delimiter of any length.

chore: abstract chunks async iterator
Abstract chunks(reader, delim) async iterator function

@kevinkassimo kevinkassimo changed the title feat(cli cmd): deno xeval prototype feat(cli cmd): deno xeval May 3, 2019

@ry

ry approved these changes May 3, 2019

Copy link
Collaborator

left a comment

LGTM

@ry ry merged commit 3608117 into denoland:master May 3, 2019

3 checks passed

Travis CI - Pull Request Build Passed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@bartlomieju bartlomieju referenced this pull request May 3, 2019

Closed

deno xargs #2257

@Alhadis

This comment has been minimized.

Copy link

commented May 4, 2019

I saw xeval, got confused by the name, consulted deno help xeval, then became even more confused by a description which makes it sound as though the input is being evaluated:

Eval a script on lines (or chunks split under delimiter)

Personally, I would have named this map. The command already resembles its equivalent JavaScript:

//  ↓↓↓             ↓↓↓        ↓    ↓↓↓↓
   input     .split(" ")  .map($ => code($));
   input | deno -d  " "    -I \$   'code($)';
//  ↑↑↑             ↑↑↑        ↑    ↑↑↑↑

However, you should consider picking a different default character, because $ is vulnerable to variable expansion by the shell. That can bite unwary users who enclose their expression within double-quotes, or novice users unaware of how single- and double-quotes differ in shell syntax.

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

@Alhadis Yeah I fully agree with your concern on $. Maybe _? (Actually quite nice since it looks like a placeholder)

About the name, xeval is created under the intention that it works like xargs but also evaluates code like eval on each line. I think your point about map sounds very interesting, though technically forEach might makes more sense for comparison. Maybe we can name it xforeach, foreach or simply each?

@ry

This comment has been minimized.

Copy link
Collaborator

commented May 4, 2019

Maybe _ is a good line variable. It even looks like a line.

(I was also worried about shell explanation with $)

@ry

This comment has been minimized.

Copy link
Collaborator

commented May 4, 2019

I’d be happy to call it map if that was the consensus?

@hayd

This comment has been minimized.

Copy link
Contributor

commented May 4, 2019

eval-lines seems more explicit, has the benefit of sitting next to eval in help/tab-completion.

👍 to _ assuming line is too verbose 😄

@Alhadis

This comment has been minimized.

Copy link

commented May 4, 2019

_ would certainly be consistent with $_ in Perl (the "current value") and shell-scripts (the value of the most recent expression). Both make sense in the context of referring to the current chunk of data. 👍

As for other names, the only other suggestions might be each or read, in reference to foreach and the POSIX shell keyword read. Both are vague and don't communicate their immediate function very well; conversely, JavaScript programmers will grok with map once they see the analogy with Array.prototype.map.

I'd also modify this command to accept anything after the first argument as a filename to use as input. This is how perl -pe works when applying an expression against each line of multiple files.

@Alhadis

This comment has been minimized.

Copy link

commented May 4, 2019

@hayd eval-lines assumes the input record separator will be a newline, and the default delimiter is actually a space. Moreover, users can pick their own delimiters, which makes the command's name completely inaccurate.

@kevinkassimo

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

I’m slightly concerned about map though since map has quite a few different other meanings (and consider something like source maps)... so I personally prefer foreach or each, matching Array.prototype.forEach()

@ry

This comment has been minimized.

Copy link
Collaborator

commented May 4, 2019

foreach sounds good to me

Arg::with_name("delim")
.long("delim")
.short("d")
.help("Set delimiter, defaults to newline")

This comment has been minimized.

Copy link
@hayd

hayd May 4, 2019

Contributor

Is it newline or whitespace? The subcommand help says "whitespace-delimited".

(IMO line sep seems more useful as a default... but okay.)

This comment has been minimized.

Copy link
@kevinkassimo

kevinkassimo May 4, 2019

Author Contributor

Ooopsy, it should be newline (though it is funny to see that xargs also claims to be white space delimited while actually working on new lines also)

This would be fixed along with other changes made to this feature as discussed above

juniarta added a commit to juniarta/deno that referenced this pull request May 8, 2019

Update (#2)
* Add --no-binary-download option to setup.py (denoland#1916)

* core: Behavior shouldn't be generic

We always pass around Box<[u8]>, and adding this generic is an
unnecessary complication.

Add deno_core_http_bench_test to test.py

sharedQueue works on deno_core_http_bench

* disable arm64 test

* docs: fix example (denoland#1936)

* jsdoc style guide update (denoland#1932)

* Fix TextDecoder for SharedArrayBuffer backed TypedArray (denoland#1940)

* core: remove reset() from SharedQueue API

* Add test style guide (denoland#1918)

* Fix a missing import in manual.md example for testing (denoland#1935)

* Typo fix in styleguide (denoland#1946)

* Fix entry point for tools/docs.py

* Remove unused tools/build_test.py file

* More permissions prompt options (denoland#1926)

* CI jobs should fail quicker on bad test_format or lint (denoland#1954)

* Integrate //core into existing code base

This disables a few tests which are broken still:
- tests/error_004_missing_module.test
- tests/error_005_missing_dynamic_import.test
- tests/error_006_import_ext_failure.test
- repl_test test_set_timeout
- repl_test test_async_op
- repl_test test_set_timeout_interlaced
- all of permission_prompt_test

* Re-order tests for sanity

* Re-enable permissions prompt tests (denoland#1957)

* Remove Eager read, write, accept (denoland#1959)

Removing this reduces tail latency in test/http_bench.ts by half.

* Re-implement init scripts in core (denoland#1958)

Re-enables arm64 CI test

* Deprecate DENO_BUILD_MODE=release

Instead use: tools/build.py --release

* Make write_gn_args.py quiet

* Rename //src/ to //cli/ (denoland#1962)

To better distinguish the deno_core crate from the executable deno,
which will now be called "the cli" internally.

* Add async version of http_util::fetch_sync_string

* Move fetch_remote_source, fetch_local_source

They do not depend on DenoDir, so should not be methods of it.

* Add fetch_remote_source_async (denoland#1972)

* Remove old Buf definition in cli.rs (denoland#1971)

* Separate behavior for the compiler isolate (denoland#1973)

* Add more async module loading function (denoland#1974)

* get_source_code_async
* fetch_module_meta_data_async
* fetch_module_meta_data_and_maybe_compile_async

* Improve pretty printing of objects

If an object has more than 5 elements, it is printed in abbeviated form
displaying only the keys. This is useful in the REPL when inspecting
large objects like the Deno namespace:

  > Deno
  { args, noColor, pid, env, exit, isTTY, execPath, chdir, cwd, File,
  open, stdin, stdout, stderr, read, write, seek, close, copy,
  toAsyncIterator, SeekMode, Buffer, readAll, mkdirSync, mkdir,
  makeTempDirSync, makeTempDir, chmodSync, chmod, removeSync, remove,
  renameSync, rename, readFileSync, readFile, readDirSync, readDir,
  copyFileSync, copyFile, readlinkSync, readlink, statSync, lstatSync,
  stat, lstat, symlinkSync, symlink, writeFileSync, writeFile, ErrorKind,
  DenoError, libdeno, permissions, revokePermission, truncateSync,
  truncate, connect, dial, listen, metrics, resources, run, Process,
  inspect, build, platform, version, Console, stringifyArgs,
  DomIterableMixin }

* pretty-print long strings

* Website and manual improvements (denoland#1967)

* v0.3.4

* Disable arm test again

getting this failure on travis:
https://travis-ci.com/denoland/deno/jobs/186518015

  $ $CARGO_TARGET_DIR/aarch64-unknown-linux-gnu/release/deno tests/002_hello.ts
  qemu: Unsupported syscall: 278
  qemu: Unsupported syscall: 278
  Compiling file:///home/travis/build/denoland/deno/tests/002_hello.ts

* Fix clippy errors and upgrade Rust to 1.33.0 in CI (denoland#1945)

* core: Allow terminating an Isolate from another thread (denoland#1982)

* Remove dead code (denoland#1983)

* Website updates (denoland#1984)

* remove `console` constructor (denoland#1985)

* Handle overflown shared queue (denoland#1992)

Fixes denoland#1988

* Add benchmark for max latency (denoland#1975)

* Update ts_library_builder (denoland#1920)

* Resolve callback moved from Behavior to mod_instantiate()  (denoland#1999)

This simplifies the Behavior trait and makes it more explicit where the
resolve callback is being made.

Also s/StartupScript/Script

* Check `Event` constructor params (denoland#1997)

* namespace reorg: libdeno and DenoCore to Deno.core (denoland#1998)

* fixed typo (denoland#2001)

* Do not close file on invalid seek mode (denoland#2004)

* Fix some typos (denoland#2005)

* remove `isTrusted`'s setter (denoland#2006)

* Fix error when JS is executed with --recompile (denoland#2007)

test case from @afinch7

* Rewrite readFile and writeFile (denoland#2000)

Using open/read/write

* Fix typo (denoland#2010)

* Check `CustomEvent` constructor params (denoland#2011)

* Clippy fixes (denoland#2009)

* Make sharedQueue overflow warning quieter (denoland#2008)

* Merge the --recompile and --reload flags (denoland#2003)

* Add Process.stderrOutput() (denoland#1828)

* v0.3.5

* third_party: upgrade rust crates

* Avoid search prototype chain (denoland#2019)

* check `EventTarget` params (denoland#2018)

* Move //libdeno to //core/libdeno (denoland#2015)

Fixes some sed errors introduced in c43cfe.

Unfortunately moving libdeno required splitting build.rs into two parts,
one for cli and one for core.

I've also removed the arm64 build - it's complicating things at this
re-org and we're not even testing it. I need to swing back to it and get
tools/test.py running for it.

* Call ninja directly from build.rs (denoland#2020)

* Rename crate deno_core to deno (denoland#2022)

* Remove deno_core_http_bench from core/Cargo.toml (denoland#2023)

So we don't have to have an optional tokio dependency. We build
deno_core_http_bench using GN anyway.

* Publish rust crate on crates.io (denoland#2024)

* third_party: add rust crate 'clap'

* Update `deno -h` output in manual (denoland#2030)

* Upgrade to TypeScript 3.4.1 (denoland#2027)

* Improve new TextDecoder().toString() (denoland#2032)

* Add web worker JS API (denoland#1993)

* Refactored the way worker polling is scheduled and errors are handled.
* Share the worker future as a Shared

* add scoop in the install (denoland#2035)

* Follow redirect location as new referrers for nested module imports (denoland#2031)

Fixes denoland#1742
Fixes denoland#2021

* Fix missing return type (denoland#2038)

* Clippy fixes

* cli: refactor compile_sync() for readability

* Add deno_core::RecursiveLoad for async module loading (denoland#2034)

* Refactor deno_core::RecursiveLoad to be more idiomatic (denoland#2034)

* web-compatibility console (denoland#2042)

* fix console.log when error has been caught (denoland#2041)

* fix toString for some web objects (denoland#2040)

* fix JSDoc (denoland#2043)

* Non-fatal compile_sync failures (denoland#2039)

And model worker resources as Stream

* fix code bug in jsdocs (denoland#2048)

* Improve docs in core (denoland#2049)

* v0.3.6

* add compile_async (denoland#2053)

* fix v0.3.6 date error (denoland#2051)

* Add worker benchmarks (denoland#2059)

* Fix tools/docs.py (denoland#2055)

* Test TCP_NODELAY in hyper_hello (denoland#2060)

* Refactor CLI flag parsing (denoland#2025)

* Fix typo in manual.md (denoland#2066)

* Fix unit_tests.py (denoland#2065)

They were silently broken in 780e72 due to flag reordering. This commit
also includes a new assert that would avoid that kind of failure in the
future.

* Flags clean up: DenoFlags::from (denoland#2068)

* use flatbuffer create functions to add fields (denoland#2046)

* Use -O3 instead of -O (denoland#2070)

As recommended in
https://www.reddit.com/r/rust/comments/balb45/why_is_hypers_max_latency_so_high_in_deno/ekck1ww/

* Add link/linkSync fs call for hardlinks (denoland#2074)

* fix `console instanceof Console` (denoland#2073)

* core: snapshot improvements (denoland#2052)

* Moves how snapshots are supplied to the Isolate. Previously they were
  given by Behavior::startup_data() but it was only called once at
  startup. It makes more sense (and simplifies Behavior) to pass it to the
  constructor of Isolate.
* Adds new libdeno type deno_snapshot instead of overloading
  deno_buf.
* Adds new libdeno method to delete snapshot deno_snapshot_delete().
* Renames deno_get_snapshot() to deno_snapshot_new().
* Makes StartupData hold references to snapshots. This was implicit when
  it previously held a deno_buf but is made explicit now. Note that
  include_bytes!() returns a &'static [u8] and we want to avoid
  copying that.

* Allow high precision performance.now() (denoland#1977)

* Merge Worker and Isolate types (denoland#2078)

Reduces generics.

* core: poll ops round robin

Also use a VecDeque to store pending ops to avoid exponential time complexity
when removing completed ops from the list.

* core: Rename Behavior to Dispatch (denoland#2082)

And rename IsolateState to ThreadSafeState.

Also make ThreadSafeState directly implement Dispatch. This is simpler.

* upgrade deno_std & add workaround prettier issue (denoland#2087)

* Silence clippy warnings and format source code

* Use correct type for `source_bytes` when buiding in check-only mode

* fix --v8-options (denoland#2093)

* Improve op dispatch (denoland#2088)

* v0.3.7

* Update `deno -h` output in manual (denoland#2099)

* fix: re-expose DomFile (denoland#2100)

* better wait for removed_from_resource_table_on_close test (denoland#2104)

* Add deno eval subcommand (denoland#2102)

* fix absolute path resolution from remote (denoland#2109)

* Upgrade deno_std (denoland#2111)

* docs: add a pixelated animation logo (denoland#2110)

* Fix silent error, add custom panic handler (denoland#2098)

This is to work around Tokio's panic recovery feature.
Ref tokio-rs/tokio#495
Ref tokio-rs/tokio#209
Ref denoland#1311
Fixes denoland#2097

* third_party: patch hyper to reduce worst-case response latency

* third_party: fix bug in hyper latency patch

* Small fixes in manual.md (denoland#2118)

* third_party: upgrade rust crates

* Add max memory benchmark (denoland#2061)

* Implement async module loading in CLI (denoland#2084)

* core/http_bench: support -D flag to enable logging

* core: run isolate tests within a task

This change is made in preparation for using FuturesUnordered to track
futures that are spawned by the isolate. FuturesUnordered sets up
notififications for every future that it finds to be not ready when
polled, which causes a crash if attempted outside of a task context.

* core: make Isolate use FuturesUnordered to track ops

Additionally, instead of polling ops in a loop until none of them are
ready, the isolate will now yield to the task system after delivering
the first batch of completed ops to the javascript side.

Although this makes performance a bit worse (about 15% fewer
requests/second on the 'deno_core_http_bench' benchmark), we feel that
the advantages are worth it:

* It resolves the extremely high worst-case latency that we were seeing
  on deno_core_http_bench, in particular when using the multi-threaded
  Tokio runtime, which would sometimes exceed a full second.

* Before this patch, the implementation of Isolate::poll() had to loop
  through all sub-futures and poll each one of them, which doesn't scale
  well as the number of futures managed by the isolate goes up. This
  could lead to poor performance when e.g. a server is servicing
  thousands of connected clients.

* Move deno_core_http_bench into examples dir (denoland#2127)

* Add cli dependency on tokio-rustls

* Fix clippy warnings

* benchmarks: fix max_memory and clean up (denoland#2137)

* Fix flaky tests (denoland#2139)

* benchmarks: improve syscall and thread count (denoland#2140)

* Fix redirects under async load (denoland#2133)

* chore: update eslint plugin (denoland#2146)

* avoid prototype builtin hasOwnProperty (denoland#2144)

* Improve test slow_never_ready_modules (denoland#2145)

* core: test Modules::deps and handle error cases better (denoland#2141)

* v0.3.8

* Make Deno/Deno.core not deletable/writable (denoland#2153)

* Exclude .png files from git auto-EOL (denoland#2161)

Fix denoland#1995 for users with the old git version

* fix typo in permissions explanation (denoland#2162)

* docs: improve clarity of the text (denoland#2165)

* Fix link to http_bench example in core README (denoland#2167)

* Refactor CLI entry point (denoland#2157)

Changes "deno --types" to "deno types"
and "deno --prefetch" to "deno prefetch"

* Fixes denoland#2033, shared queue push bug (denoland#2158)

* Fix flaky tests (denoland#2164)

* Remove dead code js/compiler_test.ts (denoland#2160)

* Issue/2170 (denoland#2175)

* Consistency using requiredArguments method

Replaced tuple length check in Headers class with requiredArguments
method.

* Consistency using requiredArguments method

Replaced tuple length check in UrlSearchParams class with
requiredArguments method.

* fmt

* Fix eslint warnings (denoland#2151)

Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: LE GOFF Vincent <g_n_s@hotmail.fr>

* Add Deno.kill(pid, signo) and process.kill(signo) (Unix only) (denoland#2177)

* Simplify logic in URLSearchParams, Buffer, Console (denoland#2174)

* symlink: Ignore type parameter on non-Windows platforms (denoland#2185)

Fixes denoland#2169

* third_party: upgrade rust crates

* Use tokio_threadpool's new panic_handler (denoland#2188)

* core: make Isolate concrete, remove Dispatch trait (denoland#2183)

Op dispatch is now dynamically dispatched, so slightly less efficient.
The immeasurable perf hit is a reasonable trade for the API simplicity
that is gained here.

* Upgrade CI to Node v12 (denoland#2193)

* Fix symlinkSyncNotImplemented (denoland#2198)

* core: Add test for snapshotting from Rust (denoland#2197)

* `URLSearchParams` deleting appended multiple (denoland#2201)

* v0.3.9

* Fix anchor links and add spinner to the benchmarks page (denoland#2205)

* Rename deno prefetch to deno fetch (denoland#2210)

* compile lib.deno_runtime.d.ts into executable (denoland#2209)

* Rename CLI flags to subcommands (denoland#2212)

* v0.3.10

* core: add Deps::to_json() (denoland#2223)

* URLSearchParams init with itself (denoland#2218)

* Use alt logos on benchmarks and manual page (denoland#2211)

* disable --version flag from deno version (denoland#2226)

* recover:  denoland#1517 Kill all pending accepts when TCP listener is closed (denoland#2224)

* fix test description for `atob` (denoland#2232)

* Revert "recover:  denoland#1517 Kill all pending accepts when TCP listener is closed (denoland#2224)" (denoland#2239)

Crashes while running wrk against
js/deps/https/deno.land/std/http/http_bench.ts

This reverts commit 972ac03.

* Disable flaky assert in killSuccess (denoland#2230)

* Add support for custom tsconfig.json (denoland#2089)

Use `--config`

* Rewrite flags.rs::parse_flags (denoland#2237)

* removes ? from URL when deleting all params (denoland#2217)

* flags: fix merge error (denoland#2244)

* Upgrade V8 to 7.6.53

* Fix max_latency benchmark on zero value (denoland#2247)

* Fix another flaky assert (denoland#2246)

* Make `atob` follow the spec (denoland#2242)

* Make error in ast_util more informative. (denoland#2252)

* fs: add Deno.utime/Deno.utimeSync (denoland#2241)

* Rename test targets (denoland#2262)

* core: remove unused function StrBufNullAllocPtr()

* Refactor zero-copy buffers for performance and to prevent memory leaks

* In order to prevent ArrayBuffers from getting garbage collected by V8,
  we used to store a v8::Persistent<ArrayBuffer> in a map. This patch
  introduces a custom ArrayBuffer allocator which doesn't use Persistent
  handles, but instead stores a pointer to the actual ArrayBuffer data
  alongside with a reference count. Since creating Persistent handles
  has quite a bit of overhead, this change significantly increases
  performance. Various HTTP server benchmarks report about 5-10% more
  requests per second than before.

* Previously the Persistent handle that prevented garbage collection had
  to be released manually, and this wasn't always done, which was
  causing memory leaks. This has been resolved by introducing a new
  `PinnedBuf` type in both Rust and C++ that automatically re-enables
  garbage collection when it goes out of scope.

* Zero-copy buffers are now correctly wrapped in an Option if there is a
  possibility that they're not present. This clears up a correctness
  issue where we were creating zero-length slices from a null pointer,
  which is against the rules.

* Async iterator for listener (denoland#2263)

* core: express op as enum (denoland#2255)

* doc: add long about messages for subcommands (denoland#2264)

Type deno <subcommand> -h to view descriptions.

* Add Request global constructor (denoland#2253)

* core: remove support for moving deno_buf ownership from C++ to JavaScript

The functionality hasn't been in use for a long time. Without this feature,
the `alloc_ptr` and `alloc_len` fields are no longer necessary.

* Work around Windows-only V8 concurrent initialization crash

This patch provides a work-around for an apparent V8 bug where
initializing multiple isolates concurrently leads to a crash on
Windows.

At the time of writing the cause of this crash is not exactly
understood, but it seems to be related to the V8 internal
function win64_unwindinfo::RegisterNonABICompliantCodeRange(),
which didn't exist in older versions of V8.

* Re-enable networking unit tests (denoland#2268)

The following tests were commented out in order to get this to go green :
- bodyMultipartFormData
- bodyURLEncodedFormData
- fetchRequestInitStringBody
- netConcurrentAccept
- netListenAsyncIterator

* core,cli: fix clippy warnings

* ci: run clippy

* Optimize read and write ops (denoland#2259)

* fixes body formData tests disabled in denoland#2268 (denoland#2274)

* Fix: deno --v8-options does not print v8 options (denoland#2277)

* add --no-fetch CLI flag to prevent remote downloads (denoland#2213)

* Bump required rustc version to 1.34.1

* ci: make the sanitize build fast again (denoland#2280)

sccache doesn't work for cache debug builds at the moment, because it
doesn't support the `-Xclang -fdebug-compilation-dir` flag that has been
added by the most recent V8 upgrade.

This patch should make the asan/lsan job on Travis CI fast again.

* fix: display "-0" for -0 (denoland#2281)

Added special handling code in js/console.ts

* Minor doc edits (denoland#2231)

* feat: support .mjs extension resolution (denoland#2283)

Removed `extmap` and added .mjs entry in `map_file_extension`.
The assert in the compiler does not need to be updated, since it is
resolving from the compiled cache instead of elsewhere (notice the .map
is asserted next to it)

* feat(cli cmd): deno xeval (denoland#2260)

* add "deno run" subcommand (denoland#2215)

* v0.3.11

* fix: deno version panic (denoland#2286)

* v0.4.0

* add warning for adblockers (denoland#2250)

* update manual.md for deno v0.4.0 (denoland#2291)

* fix: Add all permissions to deno xeval (denoland#2290)

* Add Deno.chown (denoland#2292)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.