io: change Reader interface #2591

kt3k · 2019-06-27T14:40:49Z

This PR changes the Reader interface from:

export interface Reader {
  read(p: Uint8Array): Promise<ReadResult>;
}

to:

export interface Reader {
  read(p: Uint8Array): Promise<number | EOF>;
}

I introduced EOF symbol according to @bartlomieju's comment and the discussion in denoland/std#444

closes #2384

bartlomieju · 2019-06-27T15:13:34Z

@kt3k you need to update /tools/deno_tcp.ts - it's still referencing ReadResult.eof

ry · 2019-06-27T15:16:06Z

@kt3k Cool! This is going to make v0.11 bump in std painful. We should start a PR over there before the release.

kt3k · 2019-06-27T15:40:50Z

@bartlomieju Thank you! 👍

@ry OK. I'll start it soon!

kt3k · 2019-06-28T05:18:34Z

ah, http_benchmark seems depending on std/http, std/io, etc.

error TS2694: Namespace 'Deno' has no exported member 'ReadResult'.
► file:///home/travis/build/denoland/deno/js/deps/https/deno.land/std/io/bufio.ts:7:24
7 type ReadResult = Deno.ReadResult;
                         ~~~~~~~~~~

Now I'm working on deno_std side kt3k/deno_std@fcda701

ry · 2019-06-28T05:23:47Z

@kt3k you should be able to update the submodule in js/deps/https/deno.land/std to your branch to be able to test.

piscisaureus · 2019-06-29T18:45:34Z

👍🏻👍🏻👍🏻

ry · 2019-06-30T14:54:12Z

@kt3k This looks good to me. Is this branch ready to land? It's a bit of a tricky situation with the deno_std upgrades...

kt3k · 2019-06-30T15:29:03Z

@ry
I think it's necessary to land and release this first for passing the test of deno_std's PR. In that case, the next version doesn't work with std until the deno_std's updates landed.
That's not ideal, but in my view, we don't have better way to update this kind of things with current workflow.

I think that If we have something like beta release of deno (which only developers can access) and beta branch of deno_std, we can do more graceful updates in this kind of situation.

bartlomieju · 2019-06-30T15:31:55Z

@kt3k latest Deno release will work with latest release of standard lib. I think it's fine if there's a minor break on master branches.

I think that If we have something like beta release of deno (which only developers can access) and beta branch of deno_std, we can do more graceful updates in this kind of situation.

-1 for now, since we're still not past 1.0 seems like a waste of resources

kt3k · 2019-06-30T15:59:36Z

@ry
I still have a little concern about @ts-ignore which I put in //tools/deno_http_proxy.ts. Please wait to land for a while.

@bartlomieju

-1 for now, since we're still not past 1.0 seems like a waste of resources

I agree with this. Maybe it's better to have such workflow after 1.0.

ry · 2019-07-01T13:28:55Z

tools/deno_http_proxy.ts

+  // TODO(kt3k): lib.deno_runtime.d.ts has 2 EOF types: Deno.EOF and io.EOF.
+  // They are identical symbols, and should be compatible. However typescript
+  // recognizes they are different types and the below call doesn't compile.
+  // @ts-ignore
  req.respond(resp);


That's annoying.

@kitsonk can you advise us?

@kitsonk
I have trouble with this line. The error is like the below:

lib.deno_runtime.d.ts of this branch contains two EOF unique symbol declarations. (This seems strange because in this branch EOF is defined only once, but .d.ts treat it as different types depending on where it is used.)

I tried to move EOF to several other places (js/eof.ts, js/deno.ts, etc) to avoid this, but I couldn't find a solution.

Probably doing something in deno_std like

export const EOF: unique symbol = Deno.EOF || Symbol("EOF"); export type EOF = typeof EOF;

?
(I did not follow the context of this discussion so I might be wrong)

The problem seems more subtle to me.
This change doesn't include 2 symbol definitions. EOF is defined only in io.ts. deno.ts import it and re-export it. In that case, EOF in io.ts and EOF in deno.ts should be the same thing. But lib.deno_runtime.d.ts (= deno types) includes 2 EOFs under namespace Deno and namespace io. Actually lib.deno_runtime.d.ts includes the entire contents of io.ts twice in Deno and io namespaces. (io namespace is not accessible from the user (maybe). I'm not sure why this exists. Probably a workaround for something.)

lib.deno_runtime.d.ts is created by the special tool //tools/ts_library_builder/. I suspect something goes wrong in it.

Found a workaround for this:

in //js/io.ts

// TODO: Make this symbol type export const EOF: "EOF" = "EOF"; export type EOF = typeof EOF;

This avoids the above problem.

@j-f1
I think symbol type is ideal for the type of EOF and we need to make it symbol in the future. But for now I don't see any way to declare it as symbol correctly, so I would suggest the above as an alternative path to make things forward.

Found another workaround. If we move EOF to global namespace (window), it seemed working. The diff looks like this ( kt3k@828cf3a )

@kt3k TBH I don't understand the issue, but I'm fine with that workaround (it's better than making it global). I think we should just move forward with this change and try to clean it up in later. I suspect some changes to ts_library_builder could fix this...

Could you do export const EOF: typeof Deno.EOF = Deno.EOF in io.ts?

@ry OK. I'll update the PR to use the workaround.

I suspect some changes to ts_library_builder could fix this.

I agree that this needs some changes to ts_library_builder.

@j-f1
That causes the following error:

python ../../tools/run_node.py ../../node_modules/ts-node/dist/bin.js --project /Users/kt3k/s/deno/tools/ts_library_builder/tsconfig.json --skip-ignore ../../tools/ts_library_builder/main.ts --basePath ../../ --inline ../../js/lib.web_assembly.d.ts --buildPath . --outFile gen/cli/lib/lib.deno_runtime.d.ts --silent --debug ../../js/io.ts:6:26 - error TS4025: Exported variable 'EOF' has or is using private name 'Deno'. 6 export const EOF: typeof Deno.EOF = Deno.EOF; ~~~~

kt3k · 2019-07-05T02:04:54Z

@ry
I believe this is ready to be merged now. After this released, denoland/std#527 needs to be updated soon. denoland/std#527 passes tests with custom deno of this branch on my machine, so I believe it'll be easy to update.

ry

@kt3k Sounds good. I expect we might have to adjust this a bit before the release.

I've pushed denoland/std#527 to the branch https://github.com/denoland/deno_std/tree/20190705_update_reader so that it's advertised.

Thanks for pushing through this interface change!

piscisaureus · 2019-07-05T14:51:28Z

Changing the EOF symbol to a "const string" may seem smart from a typing perspective, but it's obviously bad in practice.
Imagine you have this:

let line: string | EOF = reader.readLine();
if (line === EOF) { explode() }

What happens if the file contains a line "EOF" ?

ry

@piscisaureus makes a good point.

Can't we just put in some hack somewhere where it adds at the beginning of lib.deno_runtime.d.ts

declare Deno.EOF: unique symbol;
export type EOF = typeof Deno.EOF;

(cc @kitsonk )

kt3k · 2019-07-05T15:27:38Z

@piscisaureus @ry
OK. I see this breaks the case of string | Deno.EOF return type.

Another workaround suggestion. How about making Deno.EOF null type (as temporary solution)? This seems working and doesn't seem affecting current use cases in deno_std. (and it doesn't require the changes of ts_library_builder)

export const EOF: null = null;
export type EOF = null;

ry · 2019-07-05T22:54:19Z

@kt3k Sounds good

ry

LGTM

In #2335 a conditional was added to make sure toAsyncIterator didn't skip chunks because the reader returned data and EOF in a single call, fixing #2330. Later, in #2591, the `Reader` interface changed to `Promise<number | EOF>`. Since the reader no longer returns data and EOF in a single call, this conditional is not necessary. We can just return `{ done: true }` when we get `EOF`. Co-authored-by: Arun Srinivasan <rulfzid@gmail.com> Co-authored-by: Arun Srinivasan <rulfzid@gmail.com>

io: change Reader interface

fff520e

fix: fix the use of .eof

eb5329e

kt3k mentioned this pull request Jun 29, 2019

[WIP] io: update Reader interface denoland/std#527

Merged

std: update std to denoland/std#527

5b3ed57

kt3k force-pushed the feature/refactor-reader branch from 4bcec78 to 5b3ed57 Compare June 30, 2019 05:45

kt3k added 2 commits June 30, 2019 18:32

fix: add @ts-ignore

356e58c

docs: add comment about @ts-ignore

b3d939f

kt3k closed this Jun 30, 2019

kt3k reopened this Jun 30, 2019

ry reviewed Jul 1, 2019

View reviewed changes

io: change EOF to string literal type

103fa52

update js/deps/https/deno.land/std

cfaa8e9

ry approved these changes Jul 5, 2019

View reviewed changes

ry requested changes Jul 5, 2019

View reviewed changes

fix: change EOF to null type

2833d3e

ry approved these changes Jul 6, 2019

View reviewed changes

ry merged commit a948f9f into denoland:master Jul 6, 2019

irustm mentioned this pull request Jul 7, 2019

error TS2694: Namespace 'Deno' has no exported member 'ReadResult'. denoland/std#529

Closed

serverhiccups mentioned this pull request Jul 8, 2019

Update to Deno v0.11.0, including std libs oakserver/oak#17

Closed

kt3k mentioned this pull request Oct 31, 2019

Make EOF symbol #3244

Merged

kt3k deleted the feature/refactor-reader branch January 21, 2020 17:17

braddunbar mentioned this pull request Feb 7, 2020

toAsyncIterable: Remove unnecessary EOF check #3914

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

io: change Reader interface #2591

io: change Reader interface #2591

kt3k commented Jun 27, 2019

bartlomieju commented Jun 27, 2019

ry commented Jun 27, 2019

kt3k commented Jun 27, 2019

kt3k commented Jun 28, 2019

ry commented Jun 28, 2019

piscisaureus commented Jun 29, 2019

ry commented Jun 30, 2019

kt3k commented Jun 30, 2019

bartlomieju commented Jun 30, 2019

kt3k commented Jun 30, 2019 •

edited

Loading

ry Jul 1, 2019

kt3k Jul 1, 2019

kevinkassimo Jul 1, 2019

kt3k Jul 2, 2019 •

edited

Loading

kt3k Jul 2, 2019 •

edited

Loading

kt3k Jul 2, 2019

kt3k Jul 4, 2019 •

edited

Loading

ry Jul 4, 2019

j-f1 Jul 4, 2019

kt3k Jul 4, 2019

kt3k commented Jul 5, 2019

ry left a comment

piscisaureus commented Jul 5, 2019

ry left a comment

kt3k commented Jul 5, 2019

ry commented Jul 5, 2019 •

edited

Loading

ry left a comment

io: change Reader interface #2591

io: change Reader interface #2591

Conversation

kt3k commented Jun 27, 2019

bartlomieju commented Jun 27, 2019

ry commented Jun 27, 2019

kt3k commented Jun 27, 2019

kt3k commented Jun 28, 2019

ry commented Jun 28, 2019

piscisaureus commented Jun 29, 2019

ry commented Jun 30, 2019

kt3k commented Jun 30, 2019

bartlomieju commented Jun 30, 2019

kt3k commented Jun 30, 2019 • edited Loading

ry Jul 1, 2019

Choose a reason for hiding this comment

kt3k Jul 1, 2019

Choose a reason for hiding this comment

kevinkassimo Jul 1, 2019

Choose a reason for hiding this comment

kt3k Jul 2, 2019 • edited Loading

Choose a reason for hiding this comment

kt3k Jul 2, 2019 • edited Loading

Choose a reason for hiding this comment

kt3k Jul 2, 2019

Choose a reason for hiding this comment

kt3k Jul 4, 2019 • edited Loading

Choose a reason for hiding this comment

ry Jul 4, 2019

Choose a reason for hiding this comment

j-f1 Jul 4, 2019

Choose a reason for hiding this comment

kt3k Jul 4, 2019

Choose a reason for hiding this comment

kt3k commented Jul 5, 2019

ry left a comment

Choose a reason for hiding this comment

piscisaureus commented Jul 5, 2019

ry left a comment

Choose a reason for hiding this comment

kt3k commented Jul 5, 2019

ry commented Jul 5, 2019 • edited Loading

ry left a comment

Choose a reason for hiding this comment

kt3k commented Jun 30, 2019 •

edited

Loading

kt3k Jul 2, 2019 •

edited

Loading

kt3k Jul 2, 2019 •

edited

Loading

kt3k Jul 4, 2019 •

edited

Loading

ry commented Jul 5, 2019 •

edited

Loading