-
Notifications
You must be signed in to change notification settings - Fork 556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: migrate bookmark static pages to new http server #775
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #775 +/- ##
==========================================
+ Coverage 22.06% 24.99% +2.92%
==========================================
Files 42 45 +3
Lines 5696 5534 -162
==========================================
+ Hits 1257 1383 +126
+ Misses 4262 3959 -303
- Partials 177 192 +15 ☔ View full report in Codecov by Sentry. |
Hey @Monirzadeh, can you take a quick peek at this? Sorry that it's gotten really big, but I had to refactor several things to properly solve the main issue. Let me know if the general direction and changes seems ok and are properly understood. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WOW, this is great!
You solve a lot of things.
I left some comments and questions, I don't see any major problems. beside them we just need more unit test to cover all new functions.
I see some TODO
that we can solve them later, just point one of them to solve before merge or create an issue to not forget that.
Thanks for your time spend on the project. ❤️
6a5d8d6
to
692b157
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
everything is fine to me.
just currently we can't remove os
and create tmp file in vfs
in this part right?
if it possible remove os
from processing.go
too
err = warc.NewArchive(archivalRequest, tmpFile.Name())
if err != nil {
defer os.Remove(tmpFile.Name())
return book, false, fmt.Errorf("failed to create archive: %v", err)
}
For the external libraries that dont support it we may need to download/upload the files after working with them in a temporary file locally. As long as the files used locally are ephemeral (not used for long term storage, only temporary) we are fine. But since right now we only allow local filesystem everything should keep working as it was. |
…0@628826c by renovate (#19427) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/go-shiori/shiori](https://togithub.com/go-shiori/shiori) | minor | `v1.5.5` -> `v1.6.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>go-shiori/shiori (ghcr.io/go-shiori/shiori)</summary> ### [`v1.6.0`](https://togithub.com/go-shiori/shiori/releases/tag/v1.6.0) [Compare Source](https://togithub.com/go-shiori/shiori/compare/v1.5.5...v1.6.0) It's finally here! After some work we have started moving towards **a more usable and open API** for others to consume, with **proper session handling** (no more random logouts on server shutdowns!), improvements to **ePubs**, UX and some more! See details below for more information (important notes are the breaking changes) and please fill an issue if you see anything weird, better be safe than sorry! #### Breaking changes - The `serve` command is considered deprecated and will be removed in a future release. Right now just proxies to a new `server` command that is the one that should be used from now on. - The **server** command uses a new http backend. This **should be transparent to users** and all things should keep working as usual, but that meant refactoring some of the underlying systems too, so experiences may vary between deployments and operating systems. Please fill an issue if you see that something is not working as expected. - We are moving the API to a more stable, documented and with a proper code structure. Right now there's a mix between old and new endpoints until migration is completed. Check [the documentation](https://togithub.com/go-shiori/shiori/blob/master/docs/APIv1.md) for more information on the new API and [this roadmap filter](https://togithub.com/orgs/go-shiori/projects/2/views/11) to see progress on the API migration. - Authentication to the API now uses JWTs instead of session tokens and the **endpoint has changed to a new one**, please check the documentation mentioned above. This means that there's no longer logout issues when the server is restarted or when you log in in other computer/browser. - The `--webroot` flag **no longer modifies the routes internally**, is up to the user to proxy the routes properly to Shiori without the prefix used to serve it. That means that if you want to serve Shiori under `domain.com/shiori` you need to send the path back to Shiori without the `/shiori` prefix so routes keep working. This can be done in most reverse proxies that we're aware of. We provided [a sample configuration for Nginx](https://togithub.com/go-shiori/shiori/blob/master/docs/Configuration.md#reverse-proxies-and-the-webroot-path). PRs are welcome for other reverse proxies. - The `shiori/gopher` initial user is a full fledged user instead of being hardcoded into Shiori. If you want a new user you need to create a new owner user and then remove the `shiori` user. #### Release cadence Right now the release cadence has been slow because we made too many changes at once and we had to test and finish everything before doing this release. Plan moving forward is to iterate and release faster so we're planning smaller milestones to provide new features and fixes faster to you. I will post an update announcement when [the roadmap](https://togithub.com/orgs/go-shiori/projects/2/views/4) is reviewed. #### What's Changed - APIv1: Start working on new REST API. Refactor logic in domains. by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/497](https://togithub.com/go-shiori/shiori/pull/497) - Run legacy API and new API at the same time. by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/648](https://togithub.com/go-shiori/shiori/pull/648) - fix: docker buildx tags by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/650](https://togithub.com/go-shiori/shiori/pull/650) - Allow JWT authentication into legacy APIs by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/651](https://togithub.com/go-shiori/shiori/pull/651) - Show version in login page by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/652](https://togithub.com/go-shiori/shiori/pull/652) - fix: package-name in cleanup tag by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/655](https://togithub.com/go-shiori/shiori/pull/655) - fix: pr tag prune using other action by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/656](https://togithub.com/go-shiori/shiori/pull/656) - fix: title is never retrieved when adding bookmark by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/664](https://togithub.com/go-shiori/shiori/pull/664) - Show Shiori version on server command by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/669](https://togithub.com/go-shiori/shiori/pull/669) - chore: remove irc badge from README by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/674](https://togithub.com/go-shiori/shiori/pull/674) - fix: title overwritten if user has defined it by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/684](https://togithub.com/go-shiori/shiori/pull/684) - Proper SQLite default database and warn SHIORI_DBMS users by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/667](https://togithub.com/go-shiori/shiori/pull/667) - chore: remove verbose logger by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/685](https://togithub.com/go-shiori/shiori/pull/685) - Fix CI incorrectly tagging RC releases and disables docker builds on forks by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/686](https://togithub.com/go-shiori/shiori/pull/686) - preserve fragment in URLs ([#​315](https://togithub.com/go-shiori/shiori/issues/315)) by [@​arakimo](https://togithub.com/arakimo) in [https://github.com/go-shiori/shiori/pull/687](https://togithub.com/go-shiori/shiori/pull/687) - Swagger improvements by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/666](https://togithub.com/go-shiori/shiori/pull/666) - fix: Ensure bookmark files are correctly downloaded before deleting current ones by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/683](https://togithub.com/go-shiori/shiori/pull/683) - fix(db): handle usage of special characters in searches by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/721](https://togithub.com/go-shiori/shiori/pull/721) - fix: properly parse mysql connection string, docs update by [@​rutkai](https://togithub.com/rutkai) in [https://github.com/go-shiori/shiori/pull/730](https://togithub.com/go-shiori/shiori/pull/730) - deps: upgrade to Go 1.21 by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/698](https://togithub.com/go-shiori/shiori/pull/698) - deps: upgrade github.com/gofrs/uuid to v5 by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/736](https://togithub.com/go-shiori/shiori/pull/736) - feat: build css from less files locally by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/735](https://togithub.com/go-shiori/shiori/pull/735) - refactor: Migrate ePub generation to go-epub by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/679](https://togithub.com/go-shiori/shiori/pull/679) - chore(deps): bump the all group with 6 updates by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/738](https://togithub.com/go-shiori/shiori/pull/738) - chore(deps): bump the all group with 1 update by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/740](https://togithub.com/go-shiori/shiori/pull/740) - feat: use new JWT auth in all frontend API calls by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/743](https://togithub.com/go-shiori/shiori/pull/743) - chore(deps): bump the all group with 1 update by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/746](https://togithub.com/go-shiori/shiori/pull/746) - fix: styles-check and swag-check monitor just needed directory not project root by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/747](https://togithub.com/go-shiori/shiori/pull/747) - feat: allow resize the dialogbox for bigger/hidpi screens by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/732](https://togithub.com/go-shiori/shiori/pull/732) - feat: allow per-user settings and store them in database by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/639](https://togithub.com/go-shiori/shiori/pull/639) - fix: Remove unneeded variable and unify the way send token in header by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/763](https://togithub.com/go-shiori/shiori/pull/763) - add create ebook by default in settings by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/761](https://togithub.com/go-shiori/shiori/pull/761) - fix: Actions in overlays on mobile hard to press by [@​cbe](https://togithub.com/cbe) in [https://github.com/go-shiori/shiori/pull/759](https://togithub.com/go-shiori/shiori/pull/759) - fix: Use webp as thumbnail by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/758](https://togithub.com/go-shiori/shiori/pull/758) - Update documentation for add links to shiori in android devices from share menu by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/757](https://togithub.com/go-shiori/shiori/pull/757) - Fix typos by [@​shirayu](https://togithub.com/shirayu) in [https://github.com/go-shiori/shiori/pull/756](https://togithub.com/go-shiori/shiori/pull/756) - chore(deps): bump the all group with 1 update by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/767](https://togithub.com/go-shiori/shiori/pull/767) - refactor: migrate ebook routes by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/742](https://togithub.com/go-shiori/shiori/pull/742) - Make suggestions tapable/clickable by [@​cbe](https://togithub.com/cbe) in [https://github.com/go-shiori/shiori/pull/765](https://togithub.com/go-shiori/shiori/pull/765) - chore: frontend formatting by [@​cbe](https://togithub.com/cbe) in [https://github.com/go-shiori/shiori/pull/764](https://togithub.com/go-shiori/shiori/pull/764) - ci: add codecov reporting by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/776](https://togithub.com/go-shiori/shiori/pull/776) - deps: update go dependencies by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/777](https://togithub.com/go-shiori/shiori/pull/777) - fix typo by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/778](https://togithub.com/go-shiori/shiori/pull/778) - docs: example deployment for kubernetes by [@​JPFrancoia](https://togithub.com/JPFrancoia) in [https://github.com/go-shiori/shiori/pull/754](https://togithub.com/go-shiori/shiori/pull/754) - Add Documentation>CLI>Add bookmark by [@​LLKoder](https://togithub.com/LLKoder) in [https://github.com/go-shiori/shiori/pull/794](https://togithub.com/go-shiori/shiori/pull/794) - fix: generate coverage profile by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/797](https://togithub.com/go-shiori/shiori/pull/797) - fix: use noreferer to prevent exposing shiori instance url to archived websites by [@​istiak101](https://togithub.com/istiak101) in [https://github.com/go-shiori/shiori/pull/802](https://togithub.com/go-shiori/shiori/pull/802) - deps: upgrade by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/804](https://togithub.com/go-shiori/shiori/pull/804) - refactor: migrate bookmark static pages to new http server by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/775](https://togithub.com/go-shiori/shiori/pull/775) - Fixed lint errors after refactor by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/806](https://togithub.com/go-shiori/shiori/pull/806) - docs: updated configuration page by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/808](https://togithub.com/go-shiori/shiori/pull/808) - fix: force usage of shiori prefix for environment variables in configuration by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/807](https://togithub.com/go-shiori/shiori/pull/807) - deps: updated docker image versions by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/809](https://togithub.com/go-shiori/shiori/pull/809) - chore(deps): bump the all group with 3 updates by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/812](https://togithub.com/go-shiori/shiori/pull/812) - chore(deps): bump the all group with 3 updates by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/815](https://togithub.com/go-shiori/shiori/pull/815) - chore(deps): bump the all group with 3 updates by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/go-shiori/shiori/pull/830](https://togithub.com/go-shiori/shiori/pull/830) - fix: fixes path issues on windows by [@​Monirzadeh](https://togithub.com/Monirzadeh) in [https://github.com/go-shiori/shiori/pull/829](https://togithub.com/go-shiori/shiori/pull/829) - fix: regressions and documentation from 1.5.5 upgrade by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/837](https://togithub.com/go-shiori/shiori/pull/837) - fix: update go-epub to latest version to avoid filename errors on windows by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/840](https://togithub.com/go-shiori/shiori/pull/840) - fix: point go-epub go correct repository by [@​fmartingr](https://togithub.com/fmartingr) in [https://github.com/go-shiori/shiori/pull/842](https://togithub.com/go-shiori/shiori/pull/842) - feat: allow authentication using proxy request header by [@​PterX](https://togithub.com/PterX) in [https://github.com/go-shiori/shiori/pull/836](https://togithub.com/go-shiori/shiori/pull/836) #### New Contributors - [@​arakimo](https://togithub.com/arakimo) made their first contribution in [https://github.com/go-shiori/shiori/pull/687](https://togithub.com/go-shiori/shiori/pull/687) - [@​rutkai](https://togithub.com/rutkai) made their first contribution in [https://github.com/go-shiori/shiori/pull/730](https://togithub.com/go-shiori/shiori/pull/730) - [@​cbe](https://togithub.com/cbe) made their first contribution in [https://github.com/go-shiori/shiori/pull/759](https://togithub.com/go-shiori/shiori/pull/759) - [@​shirayu](https://togithub.com/shirayu) made their first contribution in [https://github.com/go-shiori/shiori/pull/756](https://togithub.com/go-shiori/shiori/pull/756) - [@​JPFrancoia](https://togithub.com/JPFrancoia) made their first contribution in [https://github.com/go-shiori/shiori/pull/754](https://togithub.com/go-shiori/shiori/pull/754) - [@​LLKoder](https://togithub.com/LLKoder) made their first contribution in [https://github.com/go-shiori/shiori/pull/794](https://togithub.com/go-shiori/shiori/pull/794) - [@​istiak101](https://togithub.com/istiak101) made their first contribution in [https://github.com/go-shiori/shiori/pull/802](https://togithub.com/go-shiori/shiori/pull/802) - [@​PterX](https://togithub.com/PterX) made their first contribution in [https://github.com/go-shiori/shiori/pull/836](https://togithub.com/go-shiori/shiori/pull/836) **Full Changelog**: go-shiori/shiori@v1.5.5...v1.6.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNTIuMCIsInVwZGF0ZWRJblZlciI6IjM3LjI1Mi4wIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIn0=-->
/bookmark/:id/content
to new http server/bookmark/:id/thumb
to new http server/bookmark/:id/archive
to new http servermodel.Bookmark
intomodel.BookmarkDTO
to prepare future changesBookmarkDomain
to perform operations with BookmarksHasEbook
HasArchive
GetThumbnailPath
HasThumbnail
GetBookmark
GetBookmarkContentsFromArchive
ArchiverDomain
GetBookmarkArchive
StorageDomain
afero
as filesystem abstractionFixes #751