[docs] Automatically generate documentation from JSDoc #76

xenova · 2023-04-08T11:15:15Z

Making a draft PR for comments and suggestions!

Fixes a few issues in their parser Fork: https://github.com/xenova/catharsis Original: https://github.com/hegemonic/catharsis

package.json

kungfooman · 2023-04-08T13:45:39Z

Currently the output looks like:

I dislike the dot syntax in the so called "type application": Promise.<Array.<unknown>>

Should we just remove the dots (they are applied in both jsdoc and catharsis package)?

And then about the output... right now it's just one big file, does it need to be many smaller files? I couldn't find an option for that, probably that would require some API scripting.

xenova · 2023-04-08T13:53:57Z

I dislike the dot syntax in the so called "type application": Promise.<Array.>
Should we just remove the dots (they are applied in both jsdoc and catharsis package)?

Agreed. Not too sure why they do that. Apparently it's an older syntax. That sounds like the easiest solution, yeah.

And then about the output... right now it's just one big file, does it need to be many smaller files? I couldn't find an option for that, probably that would require some API scripting.

I was looking at this yesterday, and I also couldn't find a way to do it from the CLI. Do you think we should do it on a file basis or a class basis? I'm trying to make it similar to huggingface.js' docs, which splits each class into a separate file.

This looks like it could work: https://github.com/jsdoc2md/jsdoc-to-markdown/wiki/How-to-create-one-output-file-per-class ?

kungfooman · 2023-04-08T14:58:27Z

Do you think we should do it on a file basis or a class basis? I'm trying to make it similar to huggingface.js' docs, which splits each class into a separate file.

Hm, many projects already assume that one file = one class, but I guess you want to keep them all in one huge file?

Per-class-separation sounds better to me too.

Also worked out the PR that at least catharsis is not adding its strange syntax: xenova/catharsis#1

xenova · 2023-04-08T15:12:31Z

but I guess you want to keep them all in one huge file?

For the most part, I'd actually prefer it to be separated... especially for the really large files. What about something like this:

docs/

index.md
_toctree.yml
file1.js/
- classes/
  - class1.md
  - class2.md
  - ///
- functions/
  - ...
file2.js/
...

Not too sure if "functions"/"classes" are the way to break it up more though.

takoyaro · 2023-04-08T15:22:27Z

For the most part, I'd actually prefer it to be separated... especially for the really large files. What about something like this:

docs/

index.md

_toctree.yml

file1.js/

classes/

class1.md

class2.md

///

functions/

...

file2.js/

...

Not too sure if "functions"/"classes" are the way to break it up more though.

I personally prefer that approach as well. Keeping this library's docs consistent with the HF docs will be better for the Dev experience overall.

Determine method to quantize based on supported operations

Will switch back to main doc-builder after some issues are resolved

Will remove after merged

This means the script must be run from the project's root folder

GitHub actions places files in a folder called `__w`, which causes JSDoc to ignore all source files. See jsdoc2md/jsdoc-to-markdown#137

Also adds support for: 1. `Prepend` normalizer 2. Chained decoding 3. Byte fallback

Also add unit test for decoding with and without special tokens

HuggingFaceDocBuilderDev · 2023-05-15T12:53:11Z

The documentation is not available anymore as the PR was closed or merged.

xenova added 7 commits April 7, 2023 16:51

Update post_process_object_detection JSDoc

b7d964d

Add jsdoc-to-markdown as dev dependency

c76374f

Update seq2seqLoadModel JSDoc

606de46

Update Tensor JSDoc

b127cee

Update types for tokenizers.js

a38a77d

Add npm run docs command to generate docs

eb574bb

Use custom build of catharsis for parsing JSDoc

c3f853b

Fixes a few issues in their parser Fork: https://github.com/xenova/catharsis Original: https://github.com/hegemonic/catharsis

kungfooman reviewed Apr 8, 2023

View reviewed changes

package.json Outdated Show resolved Hide resolved

xenova added 17 commits April 13, 2023 21:01

Replace CommonJS imports/exports with ES6

421aa33

Merge branch 'main' into docs

d36ba31

Update package-lock.json

9bfd126

Merge max and indexOfMax functions, and return tuple of results

cfa37d3

Move math utility functions to math_utils.js

4f158a5

[version] Set version to 1.5.0.dev

aba491e

Remove unnecessary inheritance from Function class

47b256a

Update GenerationConfig documentation

a3ed4e3

Fix bert_prepare_model_inputs when tokenizer returns array

abbbe3f

Extract test utilities to separate file

e015781

Set up basic test generator for tokenizers

9989d80

Improve conversion script

1900a42

Determine method to quantize based on supported operations

Fix types in generation.js

9f0d58c

Show warnings when running tests

b430f46

Add model cache folder (.cache) to .gitignore

93b91ca

Add loading and caching of models from the HF hub

0c63bf8

Update available environment variables

9a4b9c6

xenova added 28 commits May 13, 2023 19:59

Update env.js JSDoc

756cf66

Update README.md

19773f6

Add basic workflow files for generating docs

e17f0d3

[temp] Use custom doc-builder

0682272

Will switch back to main doc-builder after some issues are resolved

Build documentation on docs branch too

ee57c99

Will remove after merged

Update workflow paths

644a6d9

Add repo_owner to documentation build workflows

4aa4a3f

Add npm scripts for building documentation

17aa583

Use relative path for doc generation

4aec9dd

This means the script must be run from the project's root folder

Add JSDoc config file

f285691

GitHub actions places files in a folder called `__w`, which causes JSDoc to ignore all source files. See jsdoc2md/jsdoc-to-markdown#137

Update jsdoc-conf.json

8bb6fba

Update path_to_docs in GitHub workflows

518ff70

Add LlamaTokenizer (#109)

fa19890

Also adds support for: 1. `Prepend` normalizer 2. Chained decoding 3. Byte fallback

Update tokenizers and add decode unit tests

7d91549

Also add unit test for decoding with and without special tokens

Update tokenizers.test.js

6478cfa

Delete tests/tokenizers folder

2d05ec8

Update github action workflows

9c4b82d

Update documentation.yml

98e70ee

[docs] Move snippets out of source folder

dff2b3b

Add index operations to Tensor objects

13391f4

Update tasks.py

88be559

Allow negative indices for slicing

76c6377

Update package.json

ad3676f

Update package.json

6b16a55

Update README shields

4878d02

Merge branch 'main' into docs

3b86267

Update 3_examples.snippet

a2c6ee6

Update README

9640a36

xenova merged commit 07f6733 into main May 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[docs] Automatically generate documentation from JSDoc #76

[docs] Automatically generate documentation from JSDoc #76

Uh oh!

xenova commented Apr 8, 2023

Uh oh!

Uh oh!

kungfooman commented Apr 8, 2023

Uh oh!

xenova commented Apr 8, 2023 •

edited

Loading

Uh oh!

kungfooman commented Apr 8, 2023

Uh oh!

xenova commented Apr 8, 2023 •

edited

Loading

Uh oh!

takoyaro commented Apr 8, 2023

Uh oh!

HuggingFaceDocBuilderDev commented May 15, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[docs] Automatically generate documentation from JSDoc #76

[docs] Automatically generate documentation from JSDoc #76

Uh oh!

Conversation

xenova commented Apr 8, 2023

Uh oh!

Uh oh!

kungfooman commented Apr 8, 2023

Uh oh!

xenova commented Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kungfooman commented Apr 8, 2023

Uh oh!

xenova commented Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

takoyaro commented Apr 8, 2023

Uh oh!

HuggingFaceDocBuilderDev commented May 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xenova commented Apr 8, 2023 •

edited

Loading

xenova commented Apr 8, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented May 15, 2023 •

edited

Loading