Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.0.0 Public Release #1

Open
wants to merge 95 commits into
base: master
Choose a base branch
from
Open

1.0.0 Public Release #1

wants to merge 95 commits into from

Conversation

hansputera
Copy link
Owner

@hansputera hansputera commented Jul 2, 2022 •
edited
Loading

TODOs

  • Stream Worker
  • [Stream Worker] Downloader Job
  • [Stream Worker] Video Combine Job
  • Readme/Documentation.
  • Reduce memory usage when sending video, and audio data.
  • Should we receive ArrayBuffer for "combine" job?

@hansputera
feat(*): initial commit
29b08d1 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add, and setup husky
b312277 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: set engines.node to >=17.9.0
bbceee1 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add dotenv
0a95e1f 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: initial dotenv
dca759f 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: add @tinyhttp
18b23f6 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: use ESM, and add onError handler
7641567 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: setup CORS
350cd87 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add @tinyhttp/rate-limit
678db31 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: setup @tinyhttp/rate-limit
745aff4 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: set ratelimit max to 50
7665d61 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add bullmq
baa3465 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add .env
2d5c1de 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): remove bullmq
aa8eb38 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add googleapis
48afd7e 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: ignore google creds
37eea31 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: init gapi
a3a64f9 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add google_tokens
1db546a 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: disable camelcase on line 40, and 45
f0fdff8 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: init scripts
6e23477 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(scripts): init accessToken
6e49de7 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add "scripts" folder
b0f20a2 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(scripts): completed accessToken script
b7e97fa 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(gapi): init "client" first
e9a74f2 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(scripts.accessToken): import google from googleapis
a9a62b3 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: add "/" route
bc46b19 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: init api
90dd404 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add piscina, and got
62fd950 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add ow
a4b6c3e 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add ioredis
8fce38b 
Signed-off-by: GitHub <noreply@github.com>
hansputera and others added 4 commits July 3, 2022 08:08
@hansputera
chore: cleanup
ba97410 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): remove googleapis
628e024 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add Dockerfile
469b635 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add heroku.yml
ae5290c 
Signed-off-by: GitHub <noreply@github.com>
@hansputera hansputera temporarily deployed to ngeworker-1 July 3, 2022 08:24 Inactive
hansputera and others added 9 commits July 3, 2022 08:51
@hansputera
chore(Dockerfile): install pnpm
2e3b248 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: add logger
ad3fc90 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(heroku.yml): use "web" instead
1f3045e 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(workers.manager): remove "workers" map
3794ac8 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore(deps): add node-static
d2f9d40 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: add util
7310c97 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat: configure the node-static
53ee777 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(worker_list.stream): set maxQueue to 10
d716b64 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(piscina_workers.stream): return string-URL instead than Buffer
5ebe874 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
Copy link
Owner Author

hansputera commented Jul 11, 2022 •
edited
Loading

Information 5ebe874, and 2fc8928

Now, the job downloader returns the url location of the file. I think we can reduce memory usage here, it's better than returning an ArrayBuffer.

Job List

  • Stream
  • Stream-Two

@hansputera
feat(piscina_workers): return string-URL instead than ArrayBuffer
2fc8928 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
chore: add readme
02c58a0 
Signed-off-by: GitHub <noreply@github.com>
@hansputera hansputera temporarily deployed to ngeworker-1 July 11, 2022 11:53 Inactive
@hansputera
chore(deps): update all dependencies
4445083 
Signed-off-by: GitHub <noreply@github.com>
@hansputera hansputera temporarily deployed to ngeworker-1 July 17, 2022 00:14 Inactive
@hansputera
feat(piscina_workers.stream): use baseURL from @bilibili-dl/config
8916e72 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(util.getCurrentAddress): update heroku logic
8ab3afa 
Signed-off-by: GitHub <noreply@github.com>
@hansputera hansputera temporarily deployed to ngeworker-1 July 17, 2022 00:42 Inactive
@hansputera hansputera temporarily deployed to ngeworker-1 July 17, 2022 05:16 Inactive
@hansputera
Copy link
Owner Author

After looking at piscina_workers/stream-two#combine, I think we need to (only) accept the file URL rather than ArrayBuffer. Because, we need to reduce memory usage, between the client, and the server. So, we can download it (URL file), and not save it.

  1. If the host URL of the file is the same as the current project ip/domain, we should check the file in the 'downloads' folder.
  2. If the file is not in the 'downloads' folder, we must download it. (Don't keep it)
  3. After the downloading process is complete, we can continue the merging progress.

@hansputera
feat(typings.workers): change audio, and video data type
a73bf39 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(validator.mergeArgs): change args type to url
349a380 
Signed-off-by: GitHub <noreply@github.com>
@hansputera
feat(piscina_workers.stream-two): implementing #1 (comment)
573fef9 
Signed-off-by: GitHub <noreply@github.com>
@hansputera hansputera temporarily deployed to ngeworker-1 July 18, 2022 12:55 Inactive
@hansputera hansputera temporarily deployed to ngeworker-1 August 6, 2022 08:08 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@hansputera hansputera

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@hansputera