Graceful: Xorm, RepoIndexer, Cron and Others #9282

zeripath · 2019-12-07T17:21:15Z

This PR continues the current theme of #8874

We first of all move from a global gracefulManager instead using a singleton approach with single initialisation. This means that the graceful manager should not be created for simple gitea commands.

We then proceed with the gracefulisation of a few simple components:

Xorm: The default context for Xorm db requests is set to HammerTime
RepoIndexer: This maintains its own queue with database backend and therefore has safe persistence built-in
Cron Tasks
TestPullRequest: This has a db backend and re-runs through all tasks on startup so has safe persistence built-in
AddTestPullRequestTask: This is a slightly incomplete gracefulisation of this function but it is better than nothing
DeliverHooks
MirrorSync

codecov-io · 2019-12-07T18:38:00Z

Codecov Report

❗ No coverage uploaded for pull request base (master@47c24be). Click here to learn what that means.
The diff coverage is 43.04%.

@@            Coverage Diff            @@
##             master    #9282   +/-   ##
=========================================
  Coverage          ?   41.29%           
=========================================
  Files             ?      566           
  Lines             ?    73687           
  Branches          ?        0           
=========================================
  Hits              ?    30432           
  Misses            ?    39459           
  Partials          ?     3796

Impacted Files	Coverage Δ
modules/graceful/restart_unix.go	`4.44% <0%> (ø)`
cmd/web.go	`0% <0%> (ø)`
modules/graceful/server_hooks.go	`4.25% <0%> (ø)`
routers/admin/admin.go	`16.52% <0%> (ø)`
routers/install.go	`0% <0%> (ø)`
cmd/web_graceful.go	`0% <0%> (ø)`
cmd/migrate.go	`0% <0%> (ø)`
modules/ssh/ssh_graceful.go	`40% <0%> (ø)`
modules/indexer/issues/indexer.go	`63.79% <0%> (ø)`
modules/graceful/net_unix.go	`18.97% <100%> (ø)`
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47c24be...be4388c. Read the comment docs.

lunny · 2019-12-08T02:14:18Z

Xorm also support a Context(ctx) method to cancel/timeout the request to database server.

zeripath · 2019-12-08T03:40:13Z

@lunny re: Xorm. I guess the question is what should be the default context, and what should be the context for sessions.

I guess nothing should be being written to the DB after HammerTime? or do you think it should be Terminate?

services/pull/check.go:TestPullRequests run runs within the shutdown context and will respond the manager requesting it be shutdown. In future this should probably become a persitable queue with pooled workers but that restructure can wait.

Termination of syncs should take place with closure at hammer time in xorm and git

guillep2k

50% review progress. 😅

models/models.go

models/repo.go

modules/graceful/manager.go

modules/graceful/manager_unix.go

guillep2k · 2019-12-12T01:31:04Z

modules/indexer/code/bleve.go

@@ -89,10 +90,19 @@ func populateRepoIndexerAsynchronously() error {
 // should only be run when the indexer is created for the first time.
 func populateRepoIndexer(maxRepoID int64) {
 	log.Info("Populating the repo indexer with existing repositories")
+
+	isShutdown := graceful.GetManager().IsShutdown()


The logic of the repo indexer will need some rewriting. If repo IDs 100 through 50 are indexed by this loop and Gitea is shut down, repos 49 through 1 will not be indexed at the next start (this function is only called if the index was empty).

We need a persistent queue to ensure the list of remaining repos is recovered at restart.

Yup. We do.

@guillep2k persistent queue of course. That's why I wrote levelqueue. It's for simple gitea instance. Maybe we should remove channel queue and make levelqueue or other persistent queue default. For bigger instances, we can use nsq/redis.

In fact, database will store the last indexed commit id. When next commit pushed, it will index from last commit to current commit and save current commit id to database. If Gitea didn't shutdown gracefully, it will lost all indexes which will be processed on channel queue until there are some new commit pushed to the default branch of that repositories.

Don't worry I'm getting there...

So getting this truly sorted will need to be in the next breakout of #8874

I've done a partial fix - unindexed repos will be populated by populatereposindex. Dealing with the missing repo updates will need a queue I think.

zeripath@2ae0b41

modules/indexer/code/bleve.go

modules/webhook/deliver.go

services/mirror/mirror.go

guillep2k · 2019-12-12T01:48:57Z

services/mirror/mirror.go

 	}); err != nil {
 		log.Error("Update: %v", err)
 	}
 }

 // SyncMirrors checks and syncs mirrors.
-// TODO: sync more mirrors at same time.
-func SyncMirrors() {
+// FIXME: graceful: this should be a persistable queue


// FIXME: graceful: this should be a persistable queue

Does it? It looks to me that mirrors are persisted by nature, I believe. If a mirror is not up to date, it will show up at models.MirrorsIterate() next time Update(ctx) is called.

Yeah it's persistent by sticking a note in the db. I wonder if we can do something better...

guillep2k · 2019-12-14T12:00:12Z

modules/sync/unique_queue.go

+}
+
+// IDs returns the current ids in the pool
+func (q *UniqueQueue) IDs() []interface{} {


Where is this needed?

It's not needed in this PR

If it returns a snapshot of the IDs of an ever-changing queue (instead of draining it), I'd suggest renaming it to GetIDsSnapshot() or something. Otherwise, when it's used for another PR its functionality will probably be overlooked and not reviewed properly.

Heh, it's to be used once the queue is closed... Patience my friend..

guillep2k · 2019-12-14T12:11:41Z

modules/sync/unique_queue.go

 	if q.Exist(id) {
-		return true
+		return
 	}


There's a potential race condition when two goroutines attempt to add functions with the same ID and neither exists before this line, but they keep going end are both ran below. I know we've tried to avoid duplicating Exist() because of the lock, but I think the check for existance should be done inside the same lock as the rest of the function.

Damn. I didn't actually check that this code was syncing correctly. I need to stop assuming that this has been done correctly.

OK I've decided do a double check as the pool has a RWMutex.

(This has made me realise this code needs to be entirely eaten by the queue stuff in the WIP)

OK I've decided do a double check as the pool has a RWMutex.

Mutexes are pretty fast devices. The chances are high that we'll have to enter the write lock anyway. so I'd skip the read lock. There's no harm in "write locking" without doing an actual write. 😁

I mean: I'd skip the read lock. 😄

zeripath · 2019-12-14T18:39:25Z

Weird I've broken the tests somehow...

guillep2k · 2019-12-14T19:20:39Z

Weird I've broken the tests somehow...

Looks like the repos are being looked in the wrong directory:

UpdateRepoIndexer: Unable to GetDefaultBranchSha for: user21/golang, Error: chdir /go/src/code.gitea.io/gitea/integrations/gitea-integration-sqlite/gitea-repositories/user21/golang.git: no such file or directory

I think the right dir should be:

/go/src/code.gitea.io/gitea/integrations/gitea-repositories/user21/golang.git

Or has that changed lately?

zeripath · 2019-12-14T22:45:17Z

It's not that that is breaking things I think.

I suspect it's the TestPullRequests queue is full and is blocking

zeripath · 2019-12-14T22:59:34Z

Ok so populateRepoIndexer is wrong. That needs to be changed. It's just gonna jam the repo indexer queue with duplicates at present. Don't quite know what I was thinking there.

zeripath · 2019-12-15T01:10:44Z

Woo fixed it.

@guillep2k @axifive or anyone else could you just double check that this is still ok?

@guillep2k

As per @guillep2k. We check if the id is in the pool under the write lock, there's no need to pre check under the read lock

guillep2k

Yeah, I can see how it failed without the desc. LGTM now. 👍

zeripath · 2019-12-15T09:47:39Z

make lgtm work

* BREAKING * Remove unused endpoints (go-gitea#9538) * Prefix all user-generated IDs in markup (go-gitea#9477) * Enforce Gitea environment for pushes (go-gitea#8982) * Hide some user information via API if user have no enough permission (go-gitea#8655) * Move startpage/homepage translation to crowdin (go-gitea#8596) * FEATURES * Webhooks should only show sender if it makes sense (go-gitea#9601) * Provide Default messages for merges (go-gitea#9393) * Add description to labels on create issue (go-gitea#9392) * Graceful Queues: Issue Indexing and Tasks (go-gitea#9363) * Default NO_REPLY_ADDRESS to DOMAIN (go-gitea#9325) * Allow FCGI over unix sockets (go-gitea#9298) * Graceful: Xorm, RepoIndexer, Cron and Others (go-gitea#9282) * Add API for Reactions (go-gitea#9220) * Graceful: Cancel Process on monitor pages & HammerTime (go-gitea#9213) * Graceful: Allow graceful restart for unix sockets (go-gitea#9113) * Graceful: Allow graceful restart for fcgi (go-gitea#9112) * Sign protected branches (go-gitea#8993) * Add Graceful shutdown for Windows and hooks for shutdown of goroutines (go-gitea#8964) * Add Gitea icon to Emojis (go-gitea#8950) * Expand/Collapse Files and Blob Excerpt while Reviewing/Comparing code (go-gitea#8924) * Allow Custom Reactions (go-gitea#8886) * Close/reopen issues by keywords in titles and comments (go-gitea#8866) * Allow incompletely specified Time Formats (go-gitea#8816) * Prevent upload (overwrite) of lfs locked file (go-gitea#8769) * Template Repositories (go-gitea#8768) * Add /milestones endpoint (go-gitea#8733) * Make repository management section handle lfs locks (go-gitea#8726) * Respect LFS File Lock on UI (go-gitea#8719) * Add team option to grant rights for all organization repositories (go-gitea#8688) * Enabling and disabling the commit button to prevent empty commits (web editor) (go-gitea#8590) * Add setting to disable BASIC authentication (go-gitea#8586) * Expose db.SetMaxOpenConns and allow non MySQL dbs to set conn pool params (go-gitea#8528) * Allow Protected Branches to Whitelist Deploy Keys (go-gitea#8483) * Push to create repo (go-gitea#8419) * Sign merges, CRUD, Wiki and Repository initialisation with gpg key (go-gitea#7631) * Add basic repository lfs management (go-gitea#7199) * BUGFIXES * Disable remove button on repository teams when have access to all (go-gitea#9640) * Clean up old references on branch delete (go-gitea#9614) * Hide public repos owned by private orgs (go-gitea#9609) * Fix access issues on milestone and issue overview pages. (go-gitea#9603) * Fix error logged when repos qs is empty (go-gitea#9591) * Dont trigger notification twice on issue assignee change (go-gitea#9582) * Fix mirror pushed commit actions (go-gitea#9572) * Allow only specific columns to be updated on issue via API (go-gitea#9189) (go-gitea#9539) * Fix default avatar for ghost user (go-gitea#9536) * Fix download of release attachments with same name (go-gitea#9529) * Resolve deprecated INI conversion (go-gitea#9525) * Ignore empty avatars during database migration (go-gitea#9520) * Fix deleted branch isn't removed when push the branch again (go-gitea#9516) * Fix repository issues pagination bug when there are more than one label filter (go-gitea#9512) * Fix SetExpr failed (go-gitea#9506) * Remove obsolete file private/push_update.go (go-gitea#9503) * When recreating hooks, delete them first so they are recreated with the umask (go-gitea#9502) * Properly enforce gitea environment for pushes (go-gitea#9501) * Fix datarace on repo indexer queue (go-gitea#9490) * Add call to load repo prior to redirect in add/remove dependency code (go-gitea#9484) * Wrap the code indexer (go-gitea#9476) * Use Req.URL.RequestURI() to cope with FCGI urls (go-gitea#9473) * Set default ssh.minimum_key_sizes (go-gitea#9466) * Fixed issue with paging in /repos/{owner}/{repo}/git/trees/{sha} api (go-gitea#9459) * Fix wrong notification on merge (go-gitea#9450) * Issue with Migration rule v111 (go-gitea#9449) * Trigger webhook when deleting a branch after merging a PR (go-gitea#9424) * Add migration to sanitize repository original_url (go-gitea#9423) * Use OriginalURL instead of CloneAddr in migration logging (go-gitea#9418) * Push update after branch is restored (go-gitea#9416) * Fix wrong migration (go-gitea#9381) * Fix show repositories filter (go-gitea#9234) (go-gitea#9379) * Fix Slack webhook payload title generation to work with Mattermost (go-gitea#9378) * Fix double webhook for new PR (go-gitea#9375) * AuthorizedKeysCommand should not query db directly (go-gitea#9371) * Fix missed change to GetManager() (go-gitea#9361) * Fix cache problem on dashboard (go-gitea#9358) * RepoIndexer: DefaultBranch needs to be prefixed by BranchPrefix (go-gitea#9356) * Fix protected branch using IssueID (go-gitea#9348) * Fix nondeterministic behavior (go-gitea#9341) * Fix PR/issue redirects when having external tracker (go-gitea#9339) * Remove release attachments which repository has been deleted (go-gitea#9334) * Fix issue indexer not triggered when migrating a repository (go-gitea#9332) * Add SyncTags to uploader interface (go-gitea#9326) * Fix bug that release attachment files not deleted when deleting repository (go-gitea#9322) * Only sync tags after all migration release batches are completed (go-gitea#9319) * File Edit: Author/Committer interchanged (go-gitea#9297) * prebuild CSS/JS before xgo release binaries (go-gitea#9293) * Log: Ensure FLAGS=none shows no flags (go-gitea#9287) * Make Diff Detail on Pull Request Changed File UI always on Top (go-gitea#9280) * Switch CSS minifier to cssnano (go-gitea#9260) * Fix latest docker image haven't include static files. (go-gitea#9252) * Don't link wiki revision to commit (go-gitea#9244) * Change review content column to type text in db (go-gitea#9229) * Fixed topic regex pattern and added search by topic links after save (go-gitea#9219) * Add language to user API responce (go-gitea#9215) * Correct tooltip message blocked by dependencies (go-gitea#9211) * Add SimpleMDE and Fix Image Paste for Issue/Comment Editor (go-gitea#9197) * Fix panic when diff (go-gitea#9187) * Fix go-gitea#9151 - smtp logger configuration sendTos should be an array (go-gitea#9154) * Fix max length check and limit in multiple repo forms (go-gitea#9148) * Always Show Password Field on Link Account Sign-in Page (go-gitea#9147) * Properly fix displaying virtual session provider in admin panel (go-gitea#9137) * Fix race condition on indexer (go-gitea#9136) * Fix team links in HTML rendering (go-gitea#9127) * Fix race condition in ReplaceSanitizer (go-gitea#9123) * Fix what information is shown about user in API (go-gitea#9115) * Fix nil context user for template repositories (go-gitea#9099) * Hide given credentials for migrated repos. (go-gitea#9097) * Fix reCAPTCHA API URL (go-gitea#9083) * Fix password checks on admin create/edit user (go-gitea#9076) * Update golang.org/x/crypto vendor to use acme v2 (go-gitea#9056) * Ensure Written is set in GZIP ProxyResponseWriter (go-gitea#9018) * Fix wrong system notice when repository is empty (go-gitea#9010) * Fix broken link to branch from issue list (go-gitea#9003) * Fix bug when pack js (go-gitea#8992) * New review approvals shouldn't require a message (go-gitea#8991) * Shadow password correctly for session config (go-gitea#8984) * Don't send notification on pending reviews (go-gitea#8943) * Fix Notify Create Ref Error on tag creation (go-gitea#8936) * Convert EOL to UNIX-style to render MD properly (go-gitea#8925) * Migrate temp_repo.go to use git.NewCommand (go-gitea#8918) * Fix issue with user.fullname (go-gitea#8902) * Add Close() method to gogitRepository (go-gitea#8901) * Enable punctuations ending mentions (go-gitea#8889) * Fix password complexity check on registration (go-gitea#8887) * Fix require external registration password (go-gitea#8885) * Fix edit content button on migrated issue content (go-gitea#8877) * Fix permission checks for close/reopen from commit (go-gitea#8875) * Fix API Bug (fail on empty assignees) (go-gitea#8873) * Stop using git count-objects and use raw directory size for repository (go-gitea#8848) * Fix count for commit graph last page (go-gitea#8843) * Fix to close opened io resources as soon as not needed (go-gitea#8839) * Improve notification (go-gitea#8835) * Fix new user form for non-local users (go-gitea#8826) * Fix: remove duplicated signed commit icons (go-gitea#8820) * Fix (open/closed) issue count when label excluded (go-gitea#8815) * Fix SSH2 conditional in key parsing code (go-gitea#8806) * Fix 500 when edit hook (go-gitea#8782) * On windows set core.longpaths true (go-gitea#8776) * Fix commit expand button to not go to commit link (go-gitea#8745) * Avoid re-issuing redundant cross-references. (go-gitea#8734) * Fix milestone close timestamp function (go-gitea#8728) * Move webhook codes from service to webhook notification (go-gitea#8712) * Show zero lines on the line counter if the file empty (go-gitea#8700) * Fix deadline on update issue or PR via API (go-gitea#8696) * make call createMilestoneComment on newIssue func (go-gitea#8678) * Send tag create and push webhook when release created on UI (go-gitea#8671) * Prevent chrome download page as html with alt + click (go-gitea#8669) * Fix 500 when getting user as unauthenticated user (go-gitea#8653) * Graceful fixes (go-gitea#8645) * Add SubURL to redirect path (go-gitea#8632) (go-gitea#8634) * Fix extra columns from `label` table (go-gitea#8633) * Add SubURL to redirect path for transferred/renamed repos (go-gitea#8632) * Fix bug when migrate from API (go-gitea#8631) * Allow to merge if file path contains " or \ (go-gitea#8629) * Prevent removal of non-empty emoji panel following selection of duplicate (go-gitea#8609) * Ensure default gpg settings not nil and found commits have reference to repo (go-gitea#8604) * Set webhook Content-Type for application/x-www-form-urlencoded (go-gitea#8599) * Fix go-gitea#8582 by handling empty repos (go-gitea#8587) * Fix of the diff statistics view on pull request's (go-gitea#8581) * Fix bug on pull requests when transfer head repository (go-gitea#8564) * Fix template error on account page (go-gitea#8562) * Allow externalID to be UUID (go-gitea#8551) * Fix ignored error on editorconfig api (go-gitea#8550) * Fix user avatar name (go-gitea#8547) * Ensure that GitRepo is set on Empty repositories (go-gitea#8539) * Add missed close in ServeBlobLFS (go-gitea#8527) * Fix migrate mirror 500 bug (go-gitea#8526) * Fix password complexity regex for special characters (on master) (go-gitea#8525) * ENHANCEMENTS * Add a /user/login landing page option (go-gitea#9622) * Some more e-mail notification fixes (go-gitea#9596) * Add branch protection option to block merge on requested changes. (go-gitea#9592) * Add footer extra links template (go-gitea#9576) * Fix for a wrong URL in activity page of repository. (go-gitea#9571) * Update default issue template (go-gitea#9568) * Change markdown rendering from blackfriday to goldmark (go-gitea#9533) * Extend file create api with dates (go-gitea#9464) * Add ActionCommentPull action (go-gitea#9456) * Response for context on retry database connection (go-gitea#9444) * Refactor webhooks to reduce code duplication (go-gitea#9422) * update couchbase deps for new license (go-gitea#9419) * Add .ignore file for search tools (go-gitea#9417) * Remove unsued struct (go-gitea#9405) * Hide not allowed Reactions (go-gitea#9387) * Remove text from action-only webhooks (go-gitea#9377) * Move PushToBaseRepo from models to services/pull (go-gitea#9352) * Site admin could view org's members (go-gitea#9346) * Sleep longer if request speed is over github limitation (go-gitea#9335) * Refactor comment (go-gitea#9330) * Refactor code indexer (go-gitea#9313) * Remove SavePatch and generate patches on the fly (go-gitea#9302) * Move some pull request functions from models to services (go-gitea#9266) * Update JS dependencies (go-gitea#9255) * Show label list on label set (go-gitea#9251) * Redirect issue if repo has configured external tracker. (go-gitea#9247) * Allow kbd tags (go-gitea#9245) * Remove unused comment actions (go-gitea#9222) * Fixed errors logging in dump.go (go-gitea#9218) * Expose release counter to repo API response (go-gitea#9214) * Make consistent links to repository in the Slack/Mattermost notificiations (go-gitea#9205) * Expose pull request counter to repo API response (go-gitea#9202) * Extend TrackedTimes API (go-gitea#9200) * Extend StopWatch API (go-gitea#9196) * Move code indexer related code to a new package (go-gitea#9191) * Docker: ask s6 to stop all service when gitea stop (go-gitea#9171) * Variable expansion in repository templates (go-gitea#9163) * Add avatar and issue labels to template repositories (go-gitea#9149) * Show single review comments in the PR conversation tab (go-gitea#9143) * Extract createComment (go-gitea#9125) * Move PushUpdateOptions from models to repofiles (go-gitea#9124) * Alternate syntax for cross references (go-gitea#9116) * Add USE_SERVICE_WORKER setting (go-gitea#9110) * Only show part of members on orgnization dashboard and add paging for orgnization members page (go-gitea#9092) * Explore page: Add topic param to pagination (go-gitea#9077) (go-gitea#9078) * Markdown: Sanitizier Configuration (go-gitea#9075) * Add password requirement info on error (go-gitea#9074) * Allow authors to use act keywords in PR content (go-gitea#9059) * Move modules/gzip to gitea.com/macaron/gzip (go-gitea#9058) * Branch protection: Possibility to not use whitelist but allow anyone with write access (go-gitea#9055) * Context menus for comments, add quote reply (go-gitea#9043) * Update branch API endpoint to show effective branch protection. (go-gitea#9031) * Move git graph from models to modules/graph (go-gitea#9027) * Move merge actions to notification (go-gitea#9024) * Move mirror sync actions to notification (go-gitea#9022) * Add retry for migration http/https requests (go-gitea#9019) * Rewrite delivery of issue and comment mails (go-gitea#9009) * Add review comments to mail notifications (go-gitea#8996) * Refactor pull request review (go-gitea#8954) * Githook highlighter (go-gitea#8932) * Add git hooks and webhooks to template repositories; move to services (go-gitea#8926) * Only view branch or tag if it match refType requested. (go-gitea#8899) * Drop Admin attribute based on LDAP when login (continue go-gitea#1743) (go-gitea#8849) * Add additional periods to activity page (go-gitea#8829) * Update go-org to optimize code (go-gitea#8824) * Move some actions to notification/action (go-gitea#8779) * Webhook support custom proxy (go-gitea#8760) * Fix API deadline removal (go-gitea#8759) * Mark review comment as invalidated when file is deleted (go-gitea#8751) * Move pull list code to a separate file (go-gitea#8748) * Move webhook to a standalone package under modules (go-gitea#8747) * Multi repo select on issue page (go-gitea#8741) * apply exclude label on milestone issue list (go-gitea#8739) * Move issue notifications and assignee man (go-gitea#8713) * Move issue change content from models to service (go-gitea#8711) * Move issue change status from models to service (go-gitea#8691) * Move more issue assignee code from models to issue service (go-gitea#8690) * Create PR on Current Repository by Default (go-gitea#8670) * Improve Open Graph Protocol (go-gitea#8637) * Batch hook pre- and post-receive calls (go-gitea#8602) * Improve webhooks (go-gitea#8583) * Move transfer repository and rename repository on a service package and start action notification (go-gitea#8573) * Implement/Fix PR review webhooks (go-gitea#8570) * Rewrite markdown rendering to blackfriday v2 and rewrite orgmode rendering to go-org (go-gitea#8560) * Move some repositories' operations to a standalone service package (go-gitea#8557) * Allow more than 255 characters for tokens in external_login_user table (go-gitea#8554) * Move issue label operations to issue service package (go-gitea#8553) * Adjust error reporting from merge failures and use LC_ALL=C for git (go-gitea#8548) * Mail assignee when issue/pull request is assigned (go-gitea#8546) * Allow committing / adding empty files using the web ui (go-gitea#8420) (go-gitea#8532) * Move sync mirror actions to mirror service package (go-gitea#8518) * Remove arrows on numeric inputs (go-gitea#8516) * Support inline rendering of CUSTOM_URL_SCHEMES (go-gitea#8496) * Recalculate repository access only for specific user (go-gitea#8481) * Add download button for rull request diff- and patch-file (go-gitea#8470) * Add single sign-on support via SSPI on Windows (go-gitea#8463) * Move change issue title from models to issue service package (go-gitea#8456) * Add included tag on branch view (go-gitea#8449) * Make static resouces web browser cache time customized on app.ini (go-gitea#8442) * Enable Uploading/Removing Attachments When Editing an Issue/Comment (go-gitea#8426) * Add pagination to commit graph page (go-gitea#8360) * Use templates for issue e-mail subject and body (go-gitea#8329) * Move clearlabels from models to issue service (go-gitea#8326) * Move AddTestPullRequestTask to pull service package from models (go-gitea#8324) * Team permission to create repository in organization (go-gitea#8312) * Allows external rendering of other filetypes (go-gitea#8300) * Add 'Alt + click' feature to exclude labels (go-gitea#8199) * Configurable close and reopen keywords for PRs (go-gitea#8120) * Configurable URL for static resources (go-gitea#7911) * Unifies commit list in repository commit table and wiki revision page (go-gitea#7907) * Allow cross-repository dependencies on issues (go-gitea#7901) * Auto-subscribe user to repository when they commit/tag to it (go-gitea#7657) * Restore Graceful Restarting & Socket Activation (go-gitea#7274) * wiki - add 'write' 'preview' buttons to wiki edit like in issues (go-gitea#7241) * Change target branch for pull request (go-gitea#6488) * Display PR commits and diffs using base repo rather than forked (go-gitea#3648) * SECURITY * Swagger hide search field (go-gitea#9554) * Add "search" to reserved usernames (go-gitea#9063) * Switch to fomantic-ui (go-gitea#9374) * Only serve attachments when linked to issue/release and if accessible by user (go-gitea#9340) * Hide credentials when submitting migration through API (go-gitea#9102) * TESTING * Add debug option to serv to help debug problems (go-gitea#9492) * Fix the intermittent TestGPGGit failures (go-gitea#9360) * Testing: Update postgres sequences (go-gitea#9304) * Missed defer prepareTestEnv (go-gitea#9285) * Fix "data race" in testlogger (go-gitea#9159) * Yet another attempt to fix the intermittent failure of gpg git test (go-gitea#9146) * integrations: Fix Dropped Test Errors (go-gitea#9040) * services/mirror: fix dropped test errors (go-gitea#9007) * Fix intermittent GPG Git test failure (go-gitea#8968) * Update Github Migration Tests (go-gitea#8893) (go-gitea#8938) * Update heatmap fixtures to restore tests (go-gitea#8615) * TRANSLATION * Fix placeholders in the error message (go-gitea#9060) * Fix spelling of admin.users.max_repo_creation (go-gitea#8934) * Improve german translation of homepage (go-gitea#8549) * BUILD * Update gitea.com/macaron to 1.4.0 (go-gitea#9608) * Upgrade lato fonts to v16. (go-gitea#9498) * Update alpine to 3.11 (go-gitea#9440) * Upgrade blevesearch (go-gitea#9177) * Remove built js/css files from git (go-gitea#9114) * Move semantic.dropdown.custom.js to webpack (go-gitea#9064) * Check compiled files during build (go-gitea#9042) * Enable lazy-loading of gitgraph.js (go-gitea#9036) * Pack web_src/js/draw.js to public/js/index.js (go-gitea#8975) * Modernize js and use babel (go-gitea#8973) * Move index.js to web_src and use webpack to pack them (go-gitea#8598) * Restrict modules/graceful to non-windows build and shim IsChild (go-gitea#8537) * Upgrade gopkg.in/editorconfig/editorconfig-core-go.v1 (go-gitea#8501) * DOCS * Swagger info corrections (go-gitea#9441) (go-gitea#9558) * Add ALLOW_ONLY_EXTERNAL_REGISTRATION to config cheat sheet (go-gitea#8986) * Rephrase comment about RuntimeDirectory option in systemd config (go-gitea#8912) * Explicitly indicate the socket unit to use the service unit "gitea.service" (go-gitea#8804) * Adjust the must-change-password help (go-gitea#8755) * Add notice to docs for migrating from more recent versions of Gogs (go-gitea#8724) * Add explicit info about customization of homepage (go-gitea#8694) * Change external asciidoctor tool to embedded mode (go-gitea#8677) * Add Docker fail2ban configuration (go-gitea#8642) * Correct some outdated statements in the contributing guidelines (go-gitea#8612) * Basic Design guidelines (describing different parts of the code) (go-gitea#8601) * Display Gitea logo in Readme (go-gitea#8592) * Fix building from source docs to ref AppWorkPath (go-gitea#8567) * Update the provided gitea.service to mention socket activation (go-gitea#8531) * Doc added how to setup email (go-gitea#8520) * MISC * Add translatable Powered by Gitea text in footer (go-gitea#9600) * Add contrib/environment-to-ini (go-gitea#9519) * Remove unnecessary loading of settings in update hook (go-gitea#9496) * Update gitignore list (go-gitea#9437) * Update license list (go-gitea#9436) * Fix background reactions in the arc-green theme (go-gitea#9421) * Update and fix chardet import (go-gitea#9351) * Ensure LF on checkouts and in editors (go-gitea#9259) * Fixed topics margin (go-gitea#9248) * Add comment to exported function WindowsServiceName (make revive) (go-gitea#9241) * Remove empty lines on issues/pulls page (go-gitea#9232) * Fix Add Comment Button's "+" Position (go-gitea#9140) * Add first issue comment hashtag (go-gitea#9052) * Change some label colors (go-gitea#9051) * Fix double scroll in branch dropdown (go-gitea#9048) * Add comment highlight when target from url (go-gitea#9047) * Update display of reactions to issues and comments (go-gitea#9038) * Button tooltip formatting under Branches (go-gitea#9034) * Allow setting default branch via API (go-gitea#9030) * Update dashboard context for PR reviews (go-gitea#8995) * Show repository size in repo home page and settings (go-gitea#8940) * Allow to add and remove all repositories to/from team. (go-gitea#8867) * Show due date in dashboard issues list (go-gitea#8860) * Theme arc-green: reverse heatmap colors (go-gitea#8840) * Project files table style update (go-gitea#8757) * gitignore debugging file from vscode (go-gitea#8740) * Add API for Issue set Subscription (go-gitea#8729) * Make 100% width search bar (go-gitea#8710) * Update color theme for heatmap (go-gitea#8709) * Add margin to title_wip_desc (go-gitea#8705) * Improve visibility of "Pending" indicator (go-gitea#8685) * Improve accessibility of dropdown menus (go-gitea#8638) * Make /users/{username}/repos list private repos the current user has access to (go-gitea#8621) * Prevent .code-view from overriding font on icon fonts (go-gitea#8614) * Add id references on all issue events to allow internal linking (go-gitea#8608) * Upgrade xorm to v0.8.0 (go-gitea#8536) * Upgrade gopkg.in/ini.v1 (go-gitea#8500) * Update CodeMirror to version 5.49.0 (go-gitea#8381) * Wiki editor: enable side-by-side button (go-gitea#7242)

zeripath changed the title ~~Graceful more things graceful~~ Graceful: RepoIndexer, Cron and TestPullRequests Dec 7, 2019

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Dec 7, 2019

zeripath mentioned this pull request Dec 7, 2019

[WIP] Yet More Graceful #8874

Closed

21 tasks

zeripath added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Dec 7, 2019

zeripath added this to the 1.11.0 milestone Dec 7, 2019

zeripath changed the title ~~Graceful: RepoIndexer, Cron and TestPullRequests~~ Graceful: Xorm, RepoIndexer, Cron and TestPullRequests Dec 8, 2019

zeripath added 7 commits December 9, 2019 15:21

Graceful: use GetManager instead of global

5b3fa6d

Graceful: Make repo indexer shutdown gracefully

00ddf85

Graceful: Make the cron tasks graceful

b2dea35

Graceful: Make TestPullRequests shutdownable

c0ea8ef

services/pull/check.go:TestPullRequests run runs within the shutdown context and will respond the manager requesting it be shutdown. In future this should probably become a persitable queue with pooled workers but that restructure can wait.

Graceful: AddTestPullRequest run in graceful ctx

7beda2d

Graceful: SetDefaultContext for Xorm to be HammerContext

db022dd

Avoid starting graceful for migrate commands and checkout

1aafb56

zeripath force-pushed the graceful-more-things-graceful branch from ec0a6d7 to 1aafb56 Compare December 9, 2019 15:22

zeripath added 3 commits December 10, 2019 20:07

Merge branch 'master' into graceful-more-things-graceful

eae3144

Graceful: DeliverHooks now can be shutdown

7589a49

Graceful: SyncMirrors shutdown

89154f5

Termination of syncs should take place with closure at hammer time in xorm and git

zeripath changed the title ~~Graceful: Xorm, RepoIndexer, Cron and TestPullRequests~~ Graceful: Xorm, RepoIndexer, Cron and Others Dec 11, 2019

Merge branch 'master' into graceful-more-things-graceful

9b5d465

guillep2k reviewed Dec 11, 2019

View reviewed changes

guillep2k reviewed Dec 12, 2019

View reviewed changes

zeripath added 6 commits December 12, 2019 09:47

Remove unnecessary ctx check

d7d6c86

Fix hammer syncing

1a79b2f

rename manager.run to manager.start

7ce742c

adjust ctx.Done check

fa751ea

Prevent deadlock in mirror.Update on shutdown

19f2ca7

Lint doesn't permit passing in ctx nil

ef9db2d

zeripath added 5 commits December 14, 2019 10:01

oops

cb2ac9b

Better error reporting

607f2c2

fixup

a4a722f

Push TestPullRequests to the UniqueQueue and make UniqueQueue closable

a9269a5

Ensure webhook queue is also closed to prevent blockage on add here

3b97005

guillep2k reviewed Dec 14, 2019

View reviewed changes

zeripath added 2 commits December 14, 2019 12:19

Remove unnecessary channel check

c227a34

Double check that we are not trying to add id to the table again

0e38ea8

zeripath added 4 commits December 14, 2019 23:52

Sort results of getunindexrepos by descending id

244b54a

Ensure repos only added once to queue

2098dd2

Merge branch 'master' into graceful-more-things-graceful

1374ad2

D'oh

d42cf0d

zeripath and others added 2 commits December 15, 2019 01:17

Skip duplicate exist check in addfunc

be4388c

As per @guillep2k. We check if the id is in the pool under the write lock, there's no need to pre check under the read lock

Merge branch 'master' into graceful-more-things-graceful

3e79dae

guillep2k approved these changes Dec 15, 2019

View reviewed changes

Merge branch 'master' into graceful-more-things-graceful

caeecd2

zeripath merged commit e3c3b33 into go-gitea:master Dec 15, 2019

zeripath deleted the graceful-more-things-graceful branch December 15, 2019 09:51

zeripath mentioned this pull request Dec 15, 2019

Fix missed change to GetManager() #9361

Merged

lunny mentioned this pull request Dec 20, 2019

Response for context on retry database connection #9444

Merged

This was referenced Jan 9, 2020

1.11.rc1 - Stuck on checking merge conflicts #9667

Closed

Move Errored PRs out of StatusChecking #9675

Merged

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graceful: Xorm, RepoIndexer, Cron and Others #9282

Graceful: Xorm, RepoIndexer, Cron and Others #9282

zeripath commented Dec 7, 2019 •

edited

Loading

codecov-io commented Dec 7, 2019 •

edited

Loading

lunny commented Dec 8, 2019

zeripath commented Dec 8, 2019

guillep2k left a comment

guillep2k Dec 12, 2019

zeripath Dec 12, 2019

lunny Dec 12, 2019

zeripath Dec 12, 2019

zeripath Dec 12, 2019

zeripath Dec 13, 2019

zeripath Dec 13, 2019

guillep2k Dec 12, 2019

zeripath Dec 12, 2019

guillep2k Dec 14, 2019

zeripath Dec 14, 2019

guillep2k Dec 14, 2019

zeripath Dec 14, 2019

guillep2k Dec 14, 2019

zeripath Dec 14, 2019

zeripath Dec 14, 2019

zeripath Dec 14, 2019

guillep2k Dec 14, 2019

guillep2k Dec 14, 2019

zeripath commented Dec 14, 2019

guillep2k commented Dec 14, 2019

zeripath commented Dec 14, 2019

zeripath commented Dec 14, 2019

zeripath commented Dec 15, 2019

guillep2k left a comment

zeripath commented Dec 15, 2019

Graceful: Xorm, RepoIndexer, Cron and Others #9282

Graceful: Xorm, RepoIndexer, Cron and Others #9282

Conversation

zeripath commented Dec 7, 2019 • edited Loading

codecov-io commented Dec 7, 2019 • edited Loading

Codecov Report

lunny commented Dec 8, 2019

zeripath commented Dec 8, 2019

guillep2k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeripath commented Dec 14, 2019

guillep2k commented Dec 14, 2019

zeripath commented Dec 14, 2019

zeripath commented Dec 14, 2019

zeripath commented Dec 15, 2019

guillep2k left a comment

Choose a reason for hiding this comment

zeripath commented Dec 15, 2019

zeripath commented Dec 7, 2019 •

edited

Loading

codecov-io commented Dec 7, 2019 •

edited

Loading