Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly determine CSV delimiter #17459

Merged
merged 32 commits into from
Oct 30, 2021

Conversation

richmahn
Copy link
Contributor

Fixes #16558

First thing this changes is that if a Filename is passed in the RenderContext object, the extension is used:

.tsv => tab
.psv => pipe (|)

If no filename is given, or the extension is the generic ".csv", we need a way to guess the delimiter that will most likely result in the CSV table will be generated. If we try all the allowed delimiters on the first 10 lines (or less if there are less than that), the one we pick should actually work with the CSV Reader and not fail (otherwise just default to comma).

So this does away with the "best score" algorithm in favor of just getting at most 10 lines, emptying any cell values that are quoted (as they may have new lines in the text and we want the first 10 rows, not just lines, of the CSV), and trying the CSV text with each delimiter to see if rendering the CSV rows passes or not. If not, go onto the next delimiter.

If two or more delimiters actually pass, the new algorithm will pick the one that results in the most columns. Many delimiters will pass, as using tab with

a,b
c,d

will of course pass, but only have 1 column. The comma delimiter results in two columns, so it is used.

@zeripath zeripath changed the title Fixes #16558: properly determine CSV delimiter Properly determine CSV delimiter Oct 27, 2021
modules/csv/csv.go Outdated Show resolved Hide resolved
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Oct 27, 2021
@wxiaoguang
Copy link
Contributor

wxiaoguang commented Oct 28, 2021

We may need some tests to cover guessDelimiter and determineDelimiter

@lunny lunny added the type/enhancement An improvement of existing functionality label Oct 28, 2021
@richmahn
Copy link
Contributor Author

We may need some tests to cover guessDelimiter and determineDelimiter

@wxiaoguang I added tons of tests. Not sure what you are saying. You have to expand that diff since it is too large! ;)

@richmahn
Copy link
Contributor Author

richmahn commented Oct 28, 2021

@wxiaoguang Since I guess people aren't seeing the test file, csv_test.go, I shorted one test that actually deals with this issue at and where the "best score" algorithm would have failed to get the tab delimiter, but my guessDelimiter() function gets it right. It's case 12 at the bottom. Now the diff shows since it didn't actually need that much text.

https://github.com/go-gitea/gitea/pull/17459/files#diff-de6131d15bff9ea571b236e6b2d9702949f1fd3b3af4648e762e39851f90f201R352

@wxiaoguang
Copy link
Contributor

Oh yes, impressive! Now the only problem is the CI failure 😂

@richmahn
Copy link
Contributor Author

@wxiaoguang I actually knew a spelling error would get me. Forgot to run that test locally. Ok, fixed.

@codecov-commenter
Copy link

codecov-commenter commented Oct 28, 2021

Codecov Report

Merging #17459 (88a2642) into main (63c0dc8) will increase coverage by 0.03%.
The diff coverage is 92.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main   #17459      +/-   ##
==========================================
+ Coverage   45.47%   45.51%   +0.03%     
==========================================
  Files         791      791              
  Lines       88702    88717      +15     
==========================================
+ Hits        40341    40380      +39     
+ Misses      41849    41828      -21     
+ Partials     6512     6509       -3     
Impacted Files Coverage Δ
routers/web/repo/compare.go 43.52% <0.00%> (ø)
modules/csv/csv.go 100.00% <100.00%> (+22.00%) ⬆️
modules/markup/csv/csv.go 33.78% <100.00%> (ø)
models/unit.go 41.09% <0.00%> (-2.74%) ⬇️
models/issue_comment.go 51.89% <0.00%> (+0.29%) ⬆️
modules/queue/queue_bytefifo.go 59.88% <0.00%> (+0.59%) ⬆️
models/repo_list.go 79.21% <0.00%> (+0.78%) ⬆️
modules/queue/workerpool.go 51.90% <0.00%> (+3.81%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63c0dc8...88a2642. Read the comment docs.

@lafriks lafriks added this to the 1.16.0 milestone Oct 28, 2021
modules/csv/csv.go Show resolved Hide resolved
modules/csv/csv.go Show resolved Hide resolved
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Oct 29, 2021
modules/csv/csv.go Outdated Show resolved Hide resolved
modules/csv/csv.go Outdated Show resolved Hide resolved
"code.gitea.io/gitea/modules/translation"
"code.gitea.io/gitea/modules/util"
)

var quoteRegexp = regexp.MustCompile(`["'][\s\S]+?["']`)
const maxLines = 10
const guessSampleSize = 1e4 // 10k
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we count file size in decimal or binary units?
Because in binary, this number should be a bit larger (10192, or 2^13, if I'm not mistaken).
I know that oftentimes, compilers prefer powers of two as those can be optimized better. I guess that should be the case here as well?

Copy link
Contributor Author

@richmahn richmahn Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't something I added, as it was always a sample size of 10,000. So yeah, it is using 'k' here in the comments that were there before I worked on this as decimal. It's just an arbitrary sample size. I made it a constant so I can check if we may have a truncated line when getting 10 lines from that sample text.

See: https://github.com/go-gitea/gitea/blob/main/modules/csv/csv.go#L34

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just an arbitrary sample size

Yes. And I'm arguing to set it to a value that the compiler will be able to optimize better.

One small correction, though: Apparently, I thought that 2^(11+1) would be 5096, instead of 4096.
So, what I am asking for is to at best use either 1<<13 (8192) or 1<<14 (16384) bytes.
Which do you prefer?
Or do you prefer the 10000, which is slightly easier to read, but will be worse in runtime performance?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't matter whatever we choose, 1e4, 16384, 8192, all work, you won't feel any difference.

We are not writing an OS or a memory allocation library, we won't benefit from the page-aligned memory size.

We do not need to struggle on such trivial details.

modules/csv/csv.go Outdated Show resolved Hide resolved
@@ -102,3 +129,20 @@ func FormatError(err error, locale translation.Locale) (string, error) {

return "", err
}

// Looks for possible delimiters right before or after (with spaces after the former) double quotes with closing quotes
var beforeAfterQuotes = regexp.MustCompile(`([,@\t;|]{0,1}) *(?:"[^"]*")+([,@\t;|]{0,1})`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if I see that correctly, adding tabs between the separator (i.e. comma) and a quoted value, we will have an incorrectly guessed delimiter (tab)? I don't think we've seen the last of this issue with this approach.

Copy link
Contributor Author

@richmahn richmahn Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@delvh Do people add tabs after a separator? I know many people find a comma delimited csv to be easier to read with a space after the comma or semicolon:

col1, col2, col3
a, b, c

But I can't image in doing

col1, 	col2, 	col3
a, 	b, 	c

Are you suggesting to allow [\s]* instead of just a space so it gets all white space? Yet then I wonder if that could mess up when there are two tabs because no value for that cell

col1 	col2 	col3
a 	 	c

Maybe

`([,@\t;|]{0,1})\s*?(?:"[^"]*")+([,@\t;|]{0,1})`

as the regex?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just realized, the above regex I suggested would now fail on this, guessing comma instead of tab, even though the first row uses a tab:

col1	col2
he said,	"""hello"""
she said,	"""goodbye"""

Here the comma is part of the text. I think if this is the case, we should just use the guestDelimiter() function and not try to do this before/after quote as it might guess wrong. If we still use it, we should probably try it on the first row to make sure we get more than two columns returned, or maybe in guessDelimiter() we call guessFromBeforeAfterQuotes and we try that on the 10 lines first, rather than trying all delimiters and if it passes, just return it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm voting it is just best to go through the list of the delimiters and find what works for the sample 1 - 10 lines, removing all quoted strings so we have no values containing new lines. If we do this "find delimiter before/after quote" we still have to check if it works with the sample text.

modules/csv/csv.go Outdated Show resolved Hide resolved
@lunny lunny merged commit 40c8451 into go-gitea:main Oct 30, 2021
@zeripath zeripath mentioned this pull request Jan 30, 2022
zeripath added a commit to zeripath/gitea that referenced this pull request Jan 30, 2022
Frontport go-gitea#18468

Frontport changelog for 1.16, frontport 1.15.11 changelog and update config.yaml

 ## [1.16.0](https://github.com/go-gitea/gitea/releases/tag/v1.16.0) - 2022-01-30

* BREAKING
  * Remove golang vendored directory (go-gitea#18277)
  * Paginate releases page & set default page size to 10 (go-gitea#16857)
  * Only allow webhook to send requests to allowed hosts (go-gitea#17482)
* SECURITY
  * Disable content sniffing on `PlainTextBytes` (go-gitea#18359) (go-gitea#18365)
  * Only view milestones from current repo (go-gitea#18414) (go-gitea#18417)
  * Sanitize user-input on file name (go-gitea#17666)
  * Use `hostmatcher` to replace `matchlist` to improve blocking of bad hosts in Webhooks (go-gitea#17605)
* FEATURES
  * Add/update SMTP auth providers via cli (go-gitea#18197)
  * Support webauthn (go-gitea#17957)
  * Team permission allow different unit has different permission (go-gitea#17811)
  * Implement Well-Known URL for password change (go-gitea#17777)
  * Add support for ssh commit signing (go-gitea#17743)
  * Allow Loading of Diffs that are too large (go-gitea#17739)
  * Add copy button to markdown code blocks (go-gitea#17638)
  * Add .gitattribute assisted language detection to blame, diff and render (go-gitea#17590)
  * Add `PULL_LIMIT` and `PUSH_LIMIT` to cron.update_mirror task (go-gitea#17568)
  * Add Reindex buttons to repository settings page (go-gitea#17494)
  * Make SSL cipher suite configurable (go-gitea#17440)
  * Add groups scope/claim to OIDC/OAuth2 Provider (go-gitea#17367)
  * Add simple update checker to Gitea (go-gitea#17212)
  * Migrated Repository will show modifications when possible (go-gitea#17191)
  * Create pub/priv keypair for federation (go-gitea#17071)
  * Make LDAP be able to skip local 2FA (go-gitea#16954)
  * Add nodeinfo endpoint for federation purposes (go-gitea#16953)
  * Save and view issue/comment content history (go-gitea#16909)
  * Use git attributes to determine generated and vendored status for language stats and diffs (go-gitea#16773)
  * Add migrate from Codebase (go-gitea#16768)
  * Add migration from GitBucket (go-gitea#16767)
  * Add OAuth2 introspection endpoint (go-gitea#16752)
  * Add proxy settings and support for migration and webhook (go-gitea#16704)
  * Add microsoft oauth2 providers (go-gitea#16544)
  * Send registration email on user autoregistration (go-gitea#16523)
  * Defer Last Commit Info (go-gitea#16467)
  * Support unprotected file patterns (go-gitea#16395)
  * Add migrate from OneDev (go-gitea#16356)
  * Add option to update pull request by `rebase` (go-gitea#16125)
  * Add RSS/Atom feed support for user actions (go-gitea#16002)
  * Add support for corporate WeChat webhooks (go-gitea#15910)
  * Add a simple way to rename branch like gh (go-gitea#15870)
  * Add bundle download for repository (go-gitea#14538)
  * Add agit flow support in gitea (go-gitea#14295)
* API
  * Add MirrorUpdated field to Repository API type (go-gitea#18267)
  * Adjust Fork API to allow setting a custom repository name (go-gitea#18066)
  * Add API to manage repo tranfers (go-gitea#17963)
  * Add API to get file commit history (go-gitea#17652)
  * Add API to get issue/pull comments and events (timeline) (go-gitea#17403)
  * Add API to get/edit wiki (go-gitea#17278)
  * Add API for get user org permissions (go-gitea#17232)
  * Add HTML urls to notification API (go-gitea#17178)
  * Add API to get commit diff/patch (go-gitea#17095)
  * Respond with updated notifications in API (go-gitea#17064)
  * Add API to fetch git notes (go-gitea#16649)
  * Generalize list header for API (go-gitea#16551)
  * Add API Token Cache (go-gitea#16547)
  * Allow Token API calls be authorized using the reverse-proxy header (go-gitea#15119)
* ENHANCEMENTS
  * Make the height of the editor in Review Box smaller (4 lines as GitHub) (go-gitea#18319)
  * Return nicer error if trying to pull from non-existent user (go-gitea#18288)
  * Show pull link for agit pull request also (go-gitea#18235)
  * Enable partial clone by default (go-gitea#18195)
  * Added replay of webhooks (go-gitea#18191)
  * Show OAuth callback error message (go-gitea#18185)
  * Increase Salt randomness (go-gitea#18179)
  * Add MP4 as default allowed attachment type (go-gitea#18170)
  * Include folders into size cost (go-gitea#18158)
  * Remove `/email2user` endpoint (go-gitea#18127)
  * Handle invalid issues (go-gitea#18111)
  * Load EasyMDE/CodeMirror dynamically, remove RequireEasyMDE (go-gitea#18069)
  * Support open compare page directly (go-gitea#17975)
  * Prefer "Hiragino Kaku Gothic ProN" in system-ui-ja (go-gitea#17954)
  * Clean legacy SimpleMDE code (go-gitea#17926)
  * Refactor install page (db type) (go-gitea#17919)
  * Improve interface when comparing a branch which has created a pull request (go-gitea#17911)
  * Allow default branch to be inferred on compare page (go-gitea#17908)
  * Display issue/comment role even if repo archived (go-gitea#17907)
  * Always set a message-id on mails (go-gitea#17900)
  * Change `<a>` elements to underline on hover (go-gitea#17898)
  * Render issue references in file table (go-gitea#17897)
  * Handle relative unix socket paths (go-gitea#17836)
  * Move accessmode into models/perm (go-gitea#17828)
  * Fix some org style problems (go-gitea#17807)
  * Add List-Unsubscribe header (go-gitea#17804)
  * Create menus for organization pages (go-gitea#17802)
  * Switch archive URL code back to href attributes (go-gitea#17796)
  * Refactor "refs/*" string usage by using constants (go-gitea#17784)
  * Allow forks to org if you can create repos (go-gitea#17783)
  * Improve install code to avoid low-level mistakes. (go-gitea#17779)
  * Improve ellipsis buttons (go-gitea#17773)
  * Add restrict and no-user-rc to authorized_keys (go-gitea#17772)
  * Add copy Commit ID button in commits list (go-gitea#17759)
  * Make `bind` error more readable (go-gitea#17750)
  * Fix navbar on project view (go-gitea#17749)
  * More pleasantly handle broken or missing git repositories (go-gitea#17747)
  * Use `*PushUpdateOptions` as receiver (go-gitea#17724)
  * Remove unused `user` paramater (go-gitea#17723)
  * Better builtin avatar generator (go-gitea#17707)
  * Cleanup and use global style on popups (go-gitea#17674)
  * Move user/org deletion to services (go-gitea#17673)
  * Added comment for changing issue ref (go-gitea#17672)
  * Allow admins to change user avatars (go-gitea#17661)
  * Only set `data-path` once for each file in diff pages (go-gitea#17657)
  * Add icon to vscode clone link (go-gitea#17641)
  * Add download button for file viewer (go-gitea#17640)
  * Add pagination to fork list (go-gitea#17639)
  * Use a standalone struct name for Organization (go-gitea#17632)
  * Minor readability patch. (go-gitea#17627)
  * Add context support for GetUserByID (go-gitea#17602)
  * Move merge-section to `> .content` (go-gitea#17582)
  * Remove NewSession method from db.Engine interface (go-gitea#17577)
  * Move unit into models/unit/ (go-gitea#17576)
  * Restrict GetDeletedBranchByID to the repositories deleted branches (go-gitea#17570)
  * Refactor commentTags functionality (go-gitea#17558)
  * Make Repo Code Indexer an Unique Queue (go-gitea#17515)
  * Simplify Gothic to use our session store instead of creating a different store (go-gitea#17507)
  * Add settings to allow different SMTP envelope from address (go-gitea#17479)
  * Properly determine CSV delimiter (go-gitea#17459)
  * Hide label comments if labels were added and removed immediately (go-gitea#17455)
  * Tune UI alignment for nav bar notification icon, avatar image, issue label (go-gitea#17438)
  * Add appearance section in settings (go-gitea#17433)
  * Move key forms before list and add cancel button (go-gitea#17432)
  * When copying executables to the docker chmod them (go-gitea#17423)
  * Remove deprecated `extendDefaultPlugins` method of svgo (go-gitea#17399)
  * Fix the click behavior for <tr> and <td> with [data-href] (go-gitea#17388)
  * Refactor update checker to use AppState (go-gitea#17387)
  * Improve async/await usage, and sort init calls in `index.js` (go-gitea#17386)
  * Use a variable but a function for IsProd because of a slight performance increment (go-gitea#17368)
  * Frontend refactor, PascalCase to camelCase, remove unused code (go-gitea#17365)
  * Hide command line merge instructions when user can't push (go-gitea#17339)
  * Move session to models/login (go-gitea#17338)
  * Sync gitea app path for git hooks and authorized keys when starting (go-gitea#17335)
  * Make the Mirror Queue a queue (go-gitea#17326)
  * Add "Copy branch name" button to pull request page (go-gitea#17323)
  * Fix repository summary on mobile (go-gitea#17322)
  * Split `index.js` to separate files (go-gitea#17315)
  * Show direct match on top for user search (go-gitea#17303)
  * Frontend refactor: move Vue related code from `index.js` to `components` dir, and remove unused codes. (go-gitea#17301)
  * Upgrade chi to v5 (go-gitea#17298)
  * Disable form autofill (go-gitea#17291)
  * Improve behavior of "Fork" button (go-gitea#17288)
  * Open markdown image links in new window (go-gitea#17287)
  * Add hints for special Wiki pages (go-gitea#17283)
  * Move add deploy key form before the list and add a cancel button (go-gitea#17228)
  * Allow adding multiple issues to a project  (go-gitea#17226)
  * Add metrics to get issues by repository (go-gitea#17225)
  * Add specific event type to header (go-gitea#17222)
  * Redirect on project after issue created (go-gitea#17211)
  * Reference in new issue modal: dont pre-populate issue title (go-gitea#17208)
  * Always set a unique Message-ID header (go-gitea#17206)
  * Add projects and project boards in exposed metrics (go-gitea#17202)
  * Add metrics to get issues by label (go-gitea#17201)
  * Add protection to disable Gitea when run as root (go-gitea#17168)
  * Don't return binary file changes in raw PR diffs by default (go-gitea#17158)
  * Support sorting for project board issuses (go-gitea#17152)
  * Force color-adjust for markdown checkboxes (go-gitea#17146)
  * Add option to copy line permalink (go-gitea#17145)
  * Move twofactor to models/login (go-gitea#17143)
  * Multiple tokens support for migrating from github (go-gitea#17134)
  * Unify issue and PR subtitles (go-gitea#17133)
  * Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (go-gitea#17125)
  * Fix problem when database id is not increment as expected (go-gitea#17124)
  * Avatar refactor, move avatar code from `models` to `models.avatars`, remove duplicated code (go-gitea#17123)
  * Re-allow clipboard copy on non-https sites (go-gitea#17118)
  * DBContext is just a Context (go-gitea#17100)
  * Move login related structs and functions to models/login (go-gitea#17093)
  * Add SkipLocal2FA option to pam and smtp sources (go-gitea#17078)
  * Move db related basic functions to models/db (go-gitea#17075)
  * Fixes username tagging in "Reference in new issue" (go-gitea#17074)
  * Use light/dark theme based on system preference (go-gitea#17051)
  * Always emit the configuration path (go-gitea#17036)
  * Add `AbsoluteListOptions` (go-gitea#17028)
  * Use common sessioner for API and Web (go-gitea#17027)
  * Fix overflow label in small view (go-gitea#17020)
  * Report the associated filter if there is an error in LDAP (go-gitea#17014)
  * Add "new issue" btn on project (go-gitea#17001)
  * Add doctor dbconsistency check for release and attachment (go-gitea#16978)
  * Disable Fomantic's CSS tooltips (go-gitea#16974)
  * Add Cache-Control to avatar redirects (go-gitea#16973)
  * Make mirror feature more configurable (go-gitea#16957)
  * Add skip and limit to git.GetTags (go-gitea#16897)
  * Remove ParseQueueConnStr as it is unused (go-gitea#16878)
  * Remove unused Fomantic sidebar module (go-gitea#16853)
  * Allow LDAP Sources to provide Avatars (go-gitea#16851)
  * Remove Dashboard/Home button from the navbar (go-gitea#16844)
  * Use conditions but not repo ids as query condition (go-gitea#16839)
  * Add user settings key/value DB table (go-gitea#16834)
  * Add buttons to allow loading of incomplete diffs (go-gitea#16829)
  * Add information for migrate failure (go-gitea#16803)
  * Add EdDSA JWT signing algorithm (go-gitea#16786)
  * Add user status filter to admin user management page (go-gitea#16770)
  * Add Option to synchronize Admin & Restricted states from OIDC/OAuth2 along with Setting Scopes (go-gitea#16766)
  * Do not use thin scrollbars on Firefox (go-gitea#16738)
  * Download LFS in git and web workflow from minio/s3 directly (SERVE_DIRECT) (go-gitea#16731)
  * Compute proper foreground color for labels (go-gitea#16729)
  * Add edit button to wiki sidebar and footer (go-gitea#16719)
  * Fix migration svg color (go-gitea#16715)
  * Add link to vscode to repo header (go-gitea#16664)
  * Add filter by owner and team to issue/pulls search endpoint (go-gitea#16662)
  * Kanban colored boards (go-gitea#16647)
  * Allow setting X-FRAME-OPTIONS (go-gitea#16643)
  * Separate open and closed issue in metrics (go-gitea#16637)
  * Support direct comparison (git diff a..b) as well merge comparison (a…b) (go-gitea#16635)
  * Add setting to OAuth handlers to skip local 2FA authentication (go-gitea#16594)
  * Make PR merge options more intuitive (go-gitea#16582)
  * Show correct text when comparing commits on empty pull request (go-gitea#16569)
  * Pre-fill suggested New File 'name' and 'content' with Query Params (go-gitea#16556)
  * Add an abstract json layout to make it's easier to change json library (go-gitea#16528)
  * Make Mermaid.js limit configurable (go-gitea#16519)
  * Improve 2FA autofill (go-gitea#16473)
  * Add modals to Organization and Team remove/leave (go-gitea#16471)
  * Show tag name on dashboard items list (go-gitea#16466)
  * Change default cron schedules from @every 24h to @midnight (go-gitea#16431)
  * Prevent double sanitize (go-gitea#16386)
  * Replace `list.List` with slices (go-gitea#16311)
  * Add configuration option to restrict users by default (go-gitea#16256)
  * Move login out of models (go-gitea#16199)
  * Support pagination of organizations on user settings pages (go-gitea#16083)
  * Switch migration icon to svg (go-gitea#15954)
  * Add left padding for chunk header of split diff view (go-gitea#13397)
  * Allow U2F 2FA without TOTP (go-gitea#11573)
* BUGFIXES
  * GitLab reviews may not have the updated_at field set (go-gitea#18450) (go-gitea#18461)
  * Fix detection of no commits when the default branch is not master (go-gitea#18422) (go-gitea#18423)
  * Fix broken oauth2 authentication source edit page (go-gitea#18412) (go-gitea#18419)
  * Place inline diff comment dialogs on split diff in 4th and 8th columns (go-gitea#18403) (go-gitea#18404)
  * Fix restore without topic failure (go-gitea#18387) (go-gitea#18400)
  * Fix commit's time (go-gitea#18375) (go-gitea#18392)
  * Fix partial cloning a repo (go-gitea#18373) (go-gitea#18377)
  * Stop trimming preceding and suffixing spaces from editor filenames (go-gitea#18334)
  * Prevent showing webauthn error for every time visiting `/user/settings/security` (go-gitea#18386)
  * Fix mime-type detection for HTTP server (go-gitea#18370) (go-gitea#18371)
  * Stop trimming preceding and suffixing spaces from editor filenames (go-gitea#18334)
  * Restore propagation of ErrDependenciesLeft (go-gitea#18325)
  * Fix PR comments UI (go-gitea#18323)
  * Use indirect comparison when showing pull requests (go-gitea#18313)
  * Replace satori/go.uuid with gofrs/uuid (go-gitea#18311)
  * Fix commit links on compare page (go-gitea#18310)
  * Don't show double error response in git hook (go-gitea#18292)
  * Handle missing default branch better in owner/repo/branches page (go-gitea#18290)
  * Fix CheckRepoStats and reuse it during migration (go-gitea#18264)
  * Prevent underline hover on cards (go-gitea#18259)
  * Don't delete branch if other PRs with this branch are open (go-gitea#18164)
  * Require codereview to have content (go-gitea#18156)
  * Allow admin to associate missing LFS objects for repositories (go-gitea#18143)
  * When attempting to subscribe other user to issue report why access denied (go-gitea#18091)
  * Add option to convert CRLF to LF line endings for sendmail (go-gitea#18075)
  * Only create pprof files for gitea serv if explicitly asked for (go-gitea#18068)
  * Abort merge if head has been updated before pressing merge (go-gitea#18032)
  * Improve TestPatch to use git read-tree -m and implement git-merge-one-file functionality (go-gitea#18004)
  * Use JSON module instead of stdlib json (go-gitea#18003)
  * Fixed issue merged/closed wording (go-gitea#17973)
  * Return nicer error for ForcePrivate (go-gitea#17971)
  * Fix overflow in commit graph (go-gitea#17947)
  * Prevent services/mailer/mailer_test.go tests from deleteing data directory (go-gitea#17941)
  * Use disable_form_autofill on Codebase and Gitbucket (go-gitea#17936)
  * Fix a panic in NotifyCreateIssueComment (caused by string truncation) (go-gitea#17928)
  * Fix markdown URL parsing (go-gitea#17924)
  * Apply CSS Variables to all message elements (go-gitea#17920)
  * Improve checkBranchName (go-gitea#17901)
  * Update chi/middleware to chi/v5/middleware (go-gitea#17888)
  * Fix position of label color picker colors (go-gitea#17866)
  * Fix ListUnadoptedRepositories incorrect total count (go-gitea#17865)
  * Remove whitespace inside rendered code `<td>` (go-gitea#17859)
  * Make Co-committed-by and co-authored-by trailers optional (go-gitea#17848)
  * Fix value of User.IsRestricted when oauth2 user registration (go-gitea#17839)
  * Use new OneDev /milestones endpoint (go-gitea#17782)
  * Prevent deadlock in TestPersistableChannelQueue (go-gitea#17717)
  * Simplify code for writing SHA to name-rev (go-gitea#17696)
  * Fix database deadlock when update issue labels (go-gitea#17649)
  * Add warning for BIDI characters in page renders and in diffs (go-gitea#17562)
  * Fix ipv6 parsing for builtin ssh server (go-gitea#17561)
  * Multiple Escaping Improvements (go-gitea#17551)
  * Fixes go-gitea#16559 - Do not trim leading spaces for tab delimited (go-gitea#17442)
  * Show client-side error if wiki page is empty (go-gitea#17415)
  * Fix context popup error (go-gitea#17398)
  * Stop sanitizing full name in API (go-gitea#17396)
  * Fix issue close/comment buttons on mobile (go-gitea#17317)
  * Fix navbar UI (go-gitea#17235)
  * Fix problem when database id is not increment as expected (go-gitea#17229)
  * Open the DingTalk link in browser (go-gitea#17084)
  * Remove heads pointing to missing old refs (go-gitea#17076)
  * Fix commit status index problem (go-gitea#17061)
  * Handle broken references in mirror sync (go-gitea#17013)
  * Fix for create repo page layout (go-gitea#17012)
  * Improve LDAP synchronization efficiency (go-gitea#16994)
  * Add repo_id for attachment (go-gitea#16958)
  * Clean-up HookPreReceive and restore functionality for pushing non-standard refs (go-gitea#16705)
  * Remove duplicate csv import in modules/csv/csv.go (go-gitea#16631)
  * Improve SMTP authentication and Fix user creation bugs  (go-gitea#16612)
  * Fixed emoji alias not parsed in links (go-gitea#16221)
  * Calculate label URL on API  (go-gitea#16186)
* TRANSLATION
  * Fix mispelling of starred as stared (go-gitea#17465)
  * Re-separate the color translation strings (go-gitea#17390)
  * Enable Malayalam, Greek, Persian, Hungarian & Indonesian by default (go-gitea#16998)
* BUILD
  * Add lockfile-check (go-gitea#18285)
  * Don't store assets modified time into generated files (go-gitea#18193)
  * Use shadowing script for docker (go-gitea#17846)
* MISC
  * Update JS dependencies (go-gitea#17611)

Signed-off-by: Andrew Thornton <art27@cantab.net>
6543 pushed a commit that referenced this pull request Jan 30, 2022
Frontport #18468

Frontport changelog for 1.16, frontport 1.15.11 changelog and update config.yaml

 ## [1.16.0](https://github.com/go-gitea/gitea/releases/tag/v1.16.0) - 2022-01-30

* BREAKING
  * Remove golang vendored directory (#18277)
  * Paginate releases page & set default page size to 10 (#16857)
  * Only allow webhook to send requests to allowed hosts (#17482)
* SECURITY
  * Disable content sniffing on `PlainTextBytes` (#18359) (#18365)
  * Only view milestones from current repo (#18414) (#18417)
  * Sanitize user-input on file name (#17666)
  * Use `hostmatcher` to replace `matchlist` to improve blocking of bad hosts in Webhooks (#17605)
* FEATURES
  * Add/update SMTP auth providers via cli (#18197)
  * Support webauthn (#17957)
  * Team permission allow different unit has different permission (#17811)
  * Implement Well-Known URL for password change (#17777)
  * Add support for ssh commit signing (#17743)
  * Allow Loading of Diffs that are too large (#17739)
  * Add copy button to markdown code blocks (#17638)
  * Add .gitattribute assisted language detection to blame, diff and render (#17590)
  * Add `PULL_LIMIT` and `PUSH_LIMIT` to cron.update_mirror task (#17568)
  * Add Reindex buttons to repository settings page (#17494)
  * Make SSL cipher suite configurable (#17440)
  * Add groups scope/claim to OIDC/OAuth2 Provider (#17367)
  * Add simple update checker to Gitea (#17212)
  * Migrated Repository will show modifications when possible (#17191)
  * Create pub/priv keypair for federation (#17071)
  * Make LDAP be able to skip local 2FA (#16954)
  * Add nodeinfo endpoint for federation purposes (#16953)
  * Save and view issue/comment content history (#16909)
  * Use git attributes to determine generated and vendored status for language stats and diffs (#16773)
  * Add migrate from Codebase (#16768)
  * Add migration from GitBucket (#16767)
  * Add OAuth2 introspection endpoint (#16752)
  * Add proxy settings and support for migration and webhook (#16704)
  * Add microsoft oauth2 providers (#16544)
  * Send registration email on user autoregistration (#16523)
  * Defer Last Commit Info (#16467)
  * Support unprotected file patterns (#16395)
  * Add migrate from OneDev (#16356)
  * Add option to update pull request by `rebase` (#16125)
  * Add RSS/Atom feed support for user actions (#16002)
  * Add support for corporate WeChat webhooks (#15910)
  * Add a simple way to rename branch like gh (#15870)
  * Add bundle download for repository (#14538)
  * Add agit flow support in gitea (#14295)
* API
  * Add MirrorUpdated field to Repository API type (#18267)
  * Adjust Fork API to allow setting a custom repository name (#18066)
  * Add API to manage repo tranfers (#17963)
  * Add API to get file commit history (#17652)
  * Add API to get issue/pull comments and events (timeline) (#17403)
  * Add API to get/edit wiki (#17278)
  * Add API for get user org permissions (#17232)
  * Add HTML urls to notification API (#17178)
  * Add API to get commit diff/patch (#17095)
  * Respond with updated notifications in API (#17064)
  * Add API to fetch git notes (#16649)
  * Generalize list header for API (#16551)
  * Add API Token Cache (#16547)
  * Allow Token API calls be authorized using the reverse-proxy header (#15119)
* ENHANCEMENTS
  * Make the height of the editor in Review Box smaller (4 lines as GitHub) (#18319)
  * Return nicer error if trying to pull from non-existent user (#18288)
  * Show pull link for agit pull request also (#18235)
  * Enable partial clone by default (#18195)
  * Added replay of webhooks (#18191)
  * Show OAuth callback error message (#18185)
  * Increase Salt randomness (#18179)
  * Add MP4 as default allowed attachment type (#18170)
  * Include folders into size cost (#18158)
  * Remove `/email2user` endpoint (#18127)
  * Handle invalid issues (#18111)
  * Load EasyMDE/CodeMirror dynamically, remove RequireEasyMDE (#18069)
  * Support open compare page directly (#17975)
  * Prefer "Hiragino Kaku Gothic ProN" in system-ui-ja (#17954)
  * Clean legacy SimpleMDE code (#17926)
  * Refactor install page (db type) (#17919)
  * Improve interface when comparing a branch which has created a pull request (#17911)
  * Allow default branch to be inferred on compare page (#17908)
  * Display issue/comment role even if repo archived (#17907)
  * Always set a message-id on mails (#17900)
  * Change `<a>` elements to underline on hover (#17898)
  * Render issue references in file table (#17897)
  * Handle relative unix socket paths (#17836)
  * Move accessmode into models/perm (#17828)
  * Fix some org style problems (#17807)
  * Add List-Unsubscribe header (#17804)
  * Create menus for organization pages (#17802)
  * Switch archive URL code back to href attributes (#17796)
  * Refactor "refs/*" string usage by using constants (#17784)
  * Allow forks to org if you can create repos (#17783)
  * Improve install code to avoid low-level mistakes. (#17779)
  * Improve ellipsis buttons (#17773)
  * Add restrict and no-user-rc to authorized_keys (#17772)
  * Add copy Commit ID button in commits list (#17759)
  * Make `bind` error more readable (#17750)
  * Fix navbar on project view (#17749)
  * More pleasantly handle broken or missing git repositories (#17747)
  * Use `*PushUpdateOptions` as receiver (#17724)
  * Remove unused `user` paramater (#17723)
  * Better builtin avatar generator (#17707)
  * Cleanup and use global style on popups (#17674)
  * Move user/org deletion to services (#17673)
  * Added comment for changing issue ref (#17672)
  * Allow admins to change user avatars (#17661)
  * Only set `data-path` once for each file in diff pages (#17657)
  * Add icon to vscode clone link (#17641)
  * Add download button for file viewer (#17640)
  * Add pagination to fork list (#17639)
  * Use a standalone struct name for Organization (#17632)
  * Minor readability patch. (#17627)
  * Add context support for GetUserByID (#17602)
  * Move merge-section to `> .content` (#17582)
  * Remove NewSession method from db.Engine interface (#17577)
  * Move unit into models/unit/ (#17576)
  * Restrict GetDeletedBranchByID to the repositories deleted branches (#17570)
  * Refactor commentTags functionality (#17558)
  * Make Repo Code Indexer an Unique Queue (#17515)
  * Simplify Gothic to use our session store instead of creating a different store (#17507)
  * Add settings to allow different SMTP envelope from address (#17479)
  * Properly determine CSV delimiter (#17459)
  * Hide label comments if labels were added and removed immediately (#17455)
  * Tune UI alignment for nav bar notification icon, avatar image, issue label (#17438)
  * Add appearance section in settings (#17433)
  * Move key forms before list and add cancel button (#17432)
  * When copying executables to the docker chmod them (#17423)
  * Remove deprecated `extendDefaultPlugins` method of svgo (#17399)
  * Fix the click behavior for <tr> and <td> with [data-href] (#17388)
  * Refactor update checker to use AppState (#17387)
  * Improve async/await usage, and sort init calls in `index.js` (#17386)
  * Use a variable but a function for IsProd because of a slight performance increment (#17368)
  * Frontend refactor, PascalCase to camelCase, remove unused code (#17365)
  * Hide command line merge instructions when user can't push (#17339)
  * Move session to models/login (#17338)
  * Sync gitea app path for git hooks and authorized keys when starting (#17335)
  * Make the Mirror Queue a queue (#17326)
  * Add "Copy branch name" button to pull request page (#17323)
  * Fix repository summary on mobile (#17322)
  * Split `index.js` to separate files (#17315)
  * Show direct match on top for user search (#17303)
  * Frontend refactor: move Vue related code from `index.js` to `components` dir, and remove unused codes. (#17301)
  * Upgrade chi to v5 (#17298)
  * Disable form autofill (#17291)
  * Improve behavior of "Fork" button (#17288)
  * Open markdown image links in new window (#17287)
  * Add hints for special Wiki pages (#17283)
  * Move add deploy key form before the list and add a cancel button (#17228)
  * Allow adding multiple issues to a project  (#17226)
  * Add metrics to get issues by repository (#17225)
  * Add specific event type to header (#17222)
  * Redirect on project after issue created (#17211)
  * Reference in new issue modal: dont pre-populate issue title (#17208)
  * Always set a unique Message-ID header (#17206)
  * Add projects and project boards in exposed metrics (#17202)
  * Add metrics to get issues by label (#17201)
  * Add protection to disable Gitea when run as root (#17168)
  * Don't return binary file changes in raw PR diffs by default (#17158)
  * Support sorting for project board issuses (#17152)
  * Force color-adjust for markdown checkboxes (#17146)
  * Add option to copy line permalink (#17145)
  * Move twofactor to models/login (#17143)
  * Multiple tokens support for migrating from github (#17134)
  * Unify issue and PR subtitles (#17133)
  * Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125)
  * Fix problem when database id is not increment as expected (#17124)
  * Avatar refactor, move avatar code from `models` to `models.avatars`, remove duplicated code (#17123)
  * Re-allow clipboard copy on non-https sites (#17118)
  * DBContext is just a Context (#17100)
  * Move login related structs and functions to models/login (#17093)
  * Add SkipLocal2FA option to pam and smtp sources (#17078)
  * Move db related basic functions to models/db (#17075)
  * Fixes username tagging in "Reference in new issue" (#17074)
  * Use light/dark theme based on system preference (#17051)
  * Always emit the configuration path (#17036)
  * Add `AbsoluteListOptions` (#17028)
  * Use common sessioner for API and Web (#17027)
  * Fix overflow label in small view (#17020)
  * Report the associated filter if there is an error in LDAP (#17014)
  * Add "new issue" btn on project (#17001)
  * Add doctor dbconsistency check for release and attachment (#16978)
  * Disable Fomantic's CSS tooltips (#16974)
  * Add Cache-Control to avatar redirects (#16973)
  * Make mirror feature more configurable (#16957)
  * Add skip and limit to git.GetTags (#16897)
  * Remove ParseQueueConnStr as it is unused (#16878)
  * Remove unused Fomantic sidebar module (#16853)
  * Allow LDAP Sources to provide Avatars (#16851)
  * Remove Dashboard/Home button from the navbar (#16844)
  * Use conditions but not repo ids as query condition (#16839)
  * Add user settings key/value DB table (#16834)
  * Add buttons to allow loading of incomplete diffs (#16829)
  * Add information for migrate failure (#16803)
  * Add EdDSA JWT signing algorithm (#16786)
  * Add user status filter to admin user management page (#16770)
  * Add Option to synchronize Admin & Restricted states from OIDC/OAuth2 along with Setting Scopes (#16766)
  * Do not use thin scrollbars on Firefox (#16738)
  * Download LFS in git and web workflow from minio/s3 directly (SERVE_DIRECT) (#16731)
  * Compute proper foreground color for labels (#16729)
  * Add edit button to wiki sidebar and footer (#16719)
  * Fix migration svg color (#16715)
  * Add link to vscode to repo header (#16664)
  * Add filter by owner and team to issue/pulls search endpoint (#16662)
  * Kanban colored boards (#16647)
  * Allow setting X-FRAME-OPTIONS (#16643)
  * Separate open and closed issue in metrics (#16637)
  * Support direct comparison (git diff a..b) as well merge comparison (a…b) (#16635)
  * Add setting to OAuth handlers to skip local 2FA authentication (#16594)
  * Make PR merge options more intuitive (#16582)
  * Show correct text when comparing commits on empty pull request (#16569)
  * Pre-fill suggested New File 'name' and 'content' with Query Params (#16556)
  * Add an abstract json layout to make it's easier to change json library (#16528)
  * Make Mermaid.js limit configurable (#16519)
  * Improve 2FA autofill (#16473)
  * Add modals to Organization and Team remove/leave (#16471)
  * Show tag name on dashboard items list (#16466)
  * Change default cron schedules from @every 24h to @midnight (#16431)
  * Prevent double sanitize (#16386)
  * Replace `list.List` with slices (#16311)
  * Add configuration option to restrict users by default (#16256)
  * Move login out of models (#16199)
  * Support pagination of organizations on user settings pages (#16083)
  * Switch migration icon to svg (#15954)
  * Add left padding for chunk header of split diff view (#13397)
  * Allow U2F 2FA without TOTP (#11573)
* BUGFIXES
  * GitLab reviews may not have the updated_at field set (#18450) (#18461)
  * Fix detection of no commits when the default branch is not master (#18422) (#18423)
  * Fix broken oauth2 authentication source edit page (#18412) (#18419)
  * Place inline diff comment dialogs on split diff in 4th and 8th columns (#18403) (#18404)
  * Fix restore without topic failure (#18387) (#18400)
  * Fix commit's time (#18375) (#18392)
  * Fix partial cloning a repo (#18373) (#18377)
  * Stop trimming preceding and suffixing spaces from editor filenames (#18334)
  * Prevent showing webauthn error for every time visiting `/user/settings/security` (#18386)
  * Fix mime-type detection for HTTP server (#18370) (#18371)
  * Stop trimming preceding and suffixing spaces from editor filenames (#18334)
  * Restore propagation of ErrDependenciesLeft (#18325)
  * Fix PR comments UI (#18323)
  * Use indirect comparison when showing pull requests (#18313)
  * Replace satori/go.uuid with gofrs/uuid (#18311)
  * Fix commit links on compare page (#18310)
  * Don't show double error response in git hook (#18292)
  * Handle missing default branch better in owner/repo/branches page (#18290)
  * Fix CheckRepoStats and reuse it during migration (#18264)
  * Prevent underline hover on cards (#18259)
  * Don't delete branch if other PRs with this branch are open (#18164)
  * Require codereview to have content (#18156)
  * Allow admin to associate missing LFS objects for repositories (#18143)
  * When attempting to subscribe other user to issue report why access denied (#18091)
  * Add option to convert CRLF to LF line endings for sendmail (#18075)
  * Only create pprof files for gitea serv if explicitly asked for (#18068)
  * Abort merge if head has been updated before pressing merge (#18032)
  * Improve TestPatch to use git read-tree -m and implement git-merge-one-file functionality (#18004)
  * Use JSON module instead of stdlib json (#18003)
  * Fixed issue merged/closed wording (#17973)
  * Return nicer error for ForcePrivate (#17971)
  * Fix overflow in commit graph (#17947)
  * Prevent services/mailer/mailer_test.go tests from deleteing data directory (#17941)
  * Use disable_form_autofill on Codebase and Gitbucket (#17936)
  * Fix a panic in NotifyCreateIssueComment (caused by string truncation) (#17928)
  * Fix markdown URL parsing (#17924)
  * Apply CSS Variables to all message elements (#17920)
  * Improve checkBranchName (#17901)
  * Update chi/middleware to chi/v5/middleware (#17888)
  * Fix position of label color picker colors (#17866)
  * Fix ListUnadoptedRepositories incorrect total count (#17865)
  * Remove whitespace inside rendered code `<td>` (#17859)
  * Make Co-committed-by and co-authored-by trailers optional (#17848)
  * Fix value of User.IsRestricted when oauth2 user registration (#17839)
  * Use new OneDev /milestones endpoint (#17782)
  * Prevent deadlock in TestPersistableChannelQueue (#17717)
  * Simplify code for writing SHA to name-rev (#17696)
  * Fix database deadlock when update issue labels (#17649)
  * Add warning for BIDI characters in page renders and in diffs (#17562)
  * Fix ipv6 parsing for builtin ssh server (#17561)
  * Multiple Escaping Improvements (#17551)
  * Fixes #16559 - Do not trim leading spaces for tab delimited (#17442)
  * Show client-side error if wiki page is empty (#17415)
  * Fix context popup error (#17398)
  * Stop sanitizing full name in API (#17396)
  * Fix issue close/comment buttons on mobile (#17317)
  * Fix navbar UI (#17235)
  * Fix problem when database id is not increment as expected (#17229)
  * Open the DingTalk link in browser (#17084)
  * Remove heads pointing to missing old refs (#17076)
  * Fix commit status index problem (#17061)
  * Handle broken references in mirror sync (#17013)
  * Fix for create repo page layout (#17012)
  * Improve LDAP synchronization efficiency (#16994)
  * Add repo_id for attachment (#16958)
  * Clean-up HookPreReceive and restore functionality for pushing non-standard refs (#16705)
  * Remove duplicate csv import in modules/csv/csv.go (#16631)
  * Improve SMTP authentication and Fix user creation bugs  (#16612)
  * Fixed emoji alias not parsed in links (#16221)
  * Calculate label URL on API  (#16186)
* TRANSLATION
  * Fix mispelling of starred as stared (#17465)
  * Re-separate the color translation strings (#17390)
  * Enable Malayalam, Greek, Persian, Hungarian & Indonesian by default (#16998)
* BUILD
  * Add lockfile-check (#18285)
  * Don't store assets modified time into generated files (#18193)
  * Use shadowing script for docker (#17846)
* MISC
  * Update JS dependencies (#17611)

Signed-off-by: Andrew Thornton <art27@cantab.net>
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
* Fixes go-gitea#16558 CSV delimiter determiner

* Fixes go-gitea#16558 - properly determine CSV delmiiter

* Moves quoteString to a new function

* Adds big test with lots of commas for tab delimited csv

* Adds comments

* Shortens the text of the test

* Removes single quotes from regexp as only double quotes need to be searched

* Fixes spelling

* Fixes check of length as it probalby will only be 1e4, not greater

* Makes sample size a const, properly removes truncated line

* Makes sample size a const, properly removes truncated line

* Fixes comment

* Fixes comment

* tests for FormatError() function

* Adds logic to find the limiter before or after a quoted value

* Simplifies regex

* Error tests

* Error tests

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

* Adds comments

* Update modules/csv/csv.go

Co-authored-by: delvh <dev.lh@web.de>

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
Frontport go-gitea#18468

Frontport changelog for 1.16, frontport 1.15.11 changelog and update config.yaml

 ## [1.16.0](https://github.com/go-gitea/gitea/releases/tag/v1.16.0) - 2022-01-30

* BREAKING
  * Remove golang vendored directory (go-gitea#18277)
  * Paginate releases page & set default page size to 10 (go-gitea#16857)
  * Only allow webhook to send requests to allowed hosts (go-gitea#17482)
* SECURITY
  * Disable content sniffing on `PlainTextBytes` (go-gitea#18359) (go-gitea#18365)
  * Only view milestones from current repo (go-gitea#18414) (go-gitea#18417)
  * Sanitize user-input on file name (go-gitea#17666)
  * Use `hostmatcher` to replace `matchlist` to improve blocking of bad hosts in Webhooks (go-gitea#17605)
* FEATURES
  * Add/update SMTP auth providers via cli (go-gitea#18197)
  * Support webauthn (go-gitea#17957)
  * Team permission allow different unit has different permission (go-gitea#17811)
  * Implement Well-Known URL for password change (go-gitea#17777)
  * Add support for ssh commit signing (go-gitea#17743)
  * Allow Loading of Diffs that are too large (go-gitea#17739)
  * Add copy button to markdown code blocks (go-gitea#17638)
  * Add .gitattribute assisted language detection to blame, diff and render (go-gitea#17590)
  * Add `PULL_LIMIT` and `PUSH_LIMIT` to cron.update_mirror task (go-gitea#17568)
  * Add Reindex buttons to repository settings page (go-gitea#17494)
  * Make SSL cipher suite configurable (go-gitea#17440)
  * Add groups scope/claim to OIDC/OAuth2 Provider (go-gitea#17367)
  * Add simple update checker to Gitea (go-gitea#17212)
  * Migrated Repository will show modifications when possible (go-gitea#17191)
  * Create pub/priv keypair for federation (go-gitea#17071)
  * Make LDAP be able to skip local 2FA (go-gitea#16954)
  * Add nodeinfo endpoint for federation purposes (go-gitea#16953)
  * Save and view issue/comment content history (go-gitea#16909)
  * Use git attributes to determine generated and vendored status for language stats and diffs (go-gitea#16773)
  * Add migrate from Codebase (go-gitea#16768)
  * Add migration from GitBucket (go-gitea#16767)
  * Add OAuth2 introspection endpoint (go-gitea#16752)
  * Add proxy settings and support for migration and webhook (go-gitea#16704)
  * Add microsoft oauth2 providers (go-gitea#16544)
  * Send registration email on user autoregistration (go-gitea#16523)
  * Defer Last Commit Info (go-gitea#16467)
  * Support unprotected file patterns (go-gitea#16395)
  * Add migrate from OneDev (go-gitea#16356)
  * Add option to update pull request by `rebase` (go-gitea#16125)
  * Add RSS/Atom feed support for user actions (go-gitea#16002)
  * Add support for corporate WeChat webhooks (go-gitea#15910)
  * Add a simple way to rename branch like gh (go-gitea#15870)
  * Add bundle download for repository (go-gitea#14538)
  * Add agit flow support in gitea (go-gitea#14295)
* API
  * Add MirrorUpdated field to Repository API type (go-gitea#18267)
  * Adjust Fork API to allow setting a custom repository name (go-gitea#18066)
  * Add API to manage repo tranfers (go-gitea#17963)
  * Add API to get file commit history (go-gitea#17652)
  * Add API to get issue/pull comments and events (timeline) (go-gitea#17403)
  * Add API to get/edit wiki (go-gitea#17278)
  * Add API for get user org permissions (go-gitea#17232)
  * Add HTML urls to notification API (go-gitea#17178)
  * Add API to get commit diff/patch (go-gitea#17095)
  * Respond with updated notifications in API (go-gitea#17064)
  * Add API to fetch git notes (go-gitea#16649)
  * Generalize list header for API (go-gitea#16551)
  * Add API Token Cache (go-gitea#16547)
  * Allow Token API calls be authorized using the reverse-proxy header (go-gitea#15119)
* ENHANCEMENTS
  * Make the height of the editor in Review Box smaller (4 lines as GitHub) (go-gitea#18319)
  * Return nicer error if trying to pull from non-existent user (go-gitea#18288)
  * Show pull link for agit pull request also (go-gitea#18235)
  * Enable partial clone by default (go-gitea#18195)
  * Added replay of webhooks (go-gitea#18191)
  * Show OAuth callback error message (go-gitea#18185)
  * Increase Salt randomness (go-gitea#18179)
  * Add MP4 as default allowed attachment type (go-gitea#18170)
  * Include folders into size cost (go-gitea#18158)
  * Remove `/email2user` endpoint (go-gitea#18127)
  * Handle invalid issues (go-gitea#18111)
  * Load EasyMDE/CodeMirror dynamically, remove RequireEasyMDE (go-gitea#18069)
  * Support open compare page directly (go-gitea#17975)
  * Prefer "Hiragino Kaku Gothic ProN" in system-ui-ja (go-gitea#17954)
  * Clean legacy SimpleMDE code (go-gitea#17926)
  * Refactor install page (db type) (go-gitea#17919)
  * Improve interface when comparing a branch which has created a pull request (go-gitea#17911)
  * Allow default branch to be inferred on compare page (go-gitea#17908)
  * Display issue/comment role even if repo archived (go-gitea#17907)
  * Always set a message-id on mails (go-gitea#17900)
  * Change `<a>` elements to underline on hover (go-gitea#17898)
  * Render issue references in file table (go-gitea#17897)
  * Handle relative unix socket paths (go-gitea#17836)
  * Move accessmode into models/perm (go-gitea#17828)
  * Fix some org style problems (go-gitea#17807)
  * Add List-Unsubscribe header (go-gitea#17804)
  * Create menus for organization pages (go-gitea#17802)
  * Switch archive URL code back to href attributes (go-gitea#17796)
  * Refactor "refs/*" string usage by using constants (go-gitea#17784)
  * Allow forks to org if you can create repos (go-gitea#17783)
  * Improve install code to avoid low-level mistakes. (go-gitea#17779)
  * Improve ellipsis buttons (go-gitea#17773)
  * Add restrict and no-user-rc to authorized_keys (go-gitea#17772)
  * Add copy Commit ID button in commits list (go-gitea#17759)
  * Make `bind` error more readable (go-gitea#17750)
  * Fix navbar on project view (go-gitea#17749)
  * More pleasantly handle broken or missing git repositories (go-gitea#17747)
  * Use `*PushUpdateOptions` as receiver (go-gitea#17724)
  * Remove unused `user` paramater (go-gitea#17723)
  * Better builtin avatar generator (go-gitea#17707)
  * Cleanup and use global style on popups (go-gitea#17674)
  * Move user/org deletion to services (go-gitea#17673)
  * Added comment for changing issue ref (go-gitea#17672)
  * Allow admins to change user avatars (go-gitea#17661)
  * Only set `data-path` once for each file in diff pages (go-gitea#17657)
  * Add icon to vscode clone link (go-gitea#17641)
  * Add download button for file viewer (go-gitea#17640)
  * Add pagination to fork list (go-gitea#17639)
  * Use a standalone struct name for Organization (go-gitea#17632)
  * Minor readability patch. (go-gitea#17627)
  * Add context support for GetUserByID (go-gitea#17602)
  * Move merge-section to `> .content` (go-gitea#17582)
  * Remove NewSession method from db.Engine interface (go-gitea#17577)
  * Move unit into models/unit/ (go-gitea#17576)
  * Restrict GetDeletedBranchByID to the repositories deleted branches (go-gitea#17570)
  * Refactor commentTags functionality (go-gitea#17558)
  * Make Repo Code Indexer an Unique Queue (go-gitea#17515)
  * Simplify Gothic to use our session store instead of creating a different store (go-gitea#17507)
  * Add settings to allow different SMTP envelope from address (go-gitea#17479)
  * Properly determine CSV delimiter (go-gitea#17459)
  * Hide label comments if labels were added and removed immediately (go-gitea#17455)
  * Tune UI alignment for nav bar notification icon, avatar image, issue label (go-gitea#17438)
  * Add appearance section in settings (go-gitea#17433)
  * Move key forms before list and add cancel button (go-gitea#17432)
  * When copying executables to the docker chmod them (go-gitea#17423)
  * Remove deprecated `extendDefaultPlugins` method of svgo (go-gitea#17399)
  * Fix the click behavior for <tr> and <td> with [data-href] (go-gitea#17388)
  * Refactor update checker to use AppState (go-gitea#17387)
  * Improve async/await usage, and sort init calls in `index.js` (go-gitea#17386)
  * Use a variable but a function for IsProd because of a slight performance increment (go-gitea#17368)
  * Frontend refactor, PascalCase to camelCase, remove unused code (go-gitea#17365)
  * Hide command line merge instructions when user can't push (go-gitea#17339)
  * Move session to models/login (go-gitea#17338)
  * Sync gitea app path for git hooks and authorized keys when starting (go-gitea#17335)
  * Make the Mirror Queue a queue (go-gitea#17326)
  * Add "Copy branch name" button to pull request page (go-gitea#17323)
  * Fix repository summary on mobile (go-gitea#17322)
  * Split `index.js` to separate files (go-gitea#17315)
  * Show direct match on top for user search (go-gitea#17303)
  * Frontend refactor: move Vue related code from `index.js` to `components` dir, and remove unused codes. (go-gitea#17301)
  * Upgrade chi to v5 (go-gitea#17298)
  * Disable form autofill (go-gitea#17291)
  * Improve behavior of "Fork" button (go-gitea#17288)
  * Open markdown image links in new window (go-gitea#17287)
  * Add hints for special Wiki pages (go-gitea#17283)
  * Move add deploy key form before the list and add a cancel button (go-gitea#17228)
  * Allow adding multiple issues to a project  (go-gitea#17226)
  * Add metrics to get issues by repository (go-gitea#17225)
  * Add specific event type to header (go-gitea#17222)
  * Redirect on project after issue created (go-gitea#17211)
  * Reference in new issue modal: dont pre-populate issue title (go-gitea#17208)
  * Always set a unique Message-ID header (go-gitea#17206)
  * Add projects and project boards in exposed metrics (go-gitea#17202)
  * Add metrics to get issues by label (go-gitea#17201)
  * Add protection to disable Gitea when run as root (go-gitea#17168)
  * Don't return binary file changes in raw PR diffs by default (go-gitea#17158)
  * Support sorting for project board issuses (go-gitea#17152)
  * Force color-adjust for markdown checkboxes (go-gitea#17146)
  * Add option to copy line permalink (go-gitea#17145)
  * Move twofactor to models/login (go-gitea#17143)
  * Multiple tokens support for migrating from github (go-gitea#17134)
  * Unify issue and PR subtitles (go-gitea#17133)
  * Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (go-gitea#17125)
  * Fix problem when database id is not increment as expected (go-gitea#17124)
  * Avatar refactor, move avatar code from `models` to `models.avatars`, remove duplicated code (go-gitea#17123)
  * Re-allow clipboard copy on non-https sites (go-gitea#17118)
  * DBContext is just a Context (go-gitea#17100)
  * Move login related structs and functions to models/login (go-gitea#17093)
  * Add SkipLocal2FA option to pam and smtp sources (go-gitea#17078)
  * Move db related basic functions to models/db (go-gitea#17075)
  * Fixes username tagging in "Reference in new issue" (go-gitea#17074)
  * Use light/dark theme based on system preference (go-gitea#17051)
  * Always emit the configuration path (go-gitea#17036)
  * Add `AbsoluteListOptions` (go-gitea#17028)
  * Use common sessioner for API and Web (go-gitea#17027)
  * Fix overflow label in small view (go-gitea#17020)
  * Report the associated filter if there is an error in LDAP (go-gitea#17014)
  * Add "new issue" btn on project (go-gitea#17001)
  * Add doctor dbconsistency check for release and attachment (go-gitea#16978)
  * Disable Fomantic's CSS tooltips (go-gitea#16974)
  * Add Cache-Control to avatar redirects (go-gitea#16973)
  * Make mirror feature more configurable (go-gitea#16957)
  * Add skip and limit to git.GetTags (go-gitea#16897)
  * Remove ParseQueueConnStr as it is unused (go-gitea#16878)
  * Remove unused Fomantic sidebar module (go-gitea#16853)
  * Allow LDAP Sources to provide Avatars (go-gitea#16851)
  * Remove Dashboard/Home button from the navbar (go-gitea#16844)
  * Use conditions but not repo ids as query condition (go-gitea#16839)
  * Add user settings key/value DB table (go-gitea#16834)
  * Add buttons to allow loading of incomplete diffs (go-gitea#16829)
  * Add information for migrate failure (go-gitea#16803)
  * Add EdDSA JWT signing algorithm (go-gitea#16786)
  * Add user status filter to admin user management page (go-gitea#16770)
  * Add Option to synchronize Admin & Restricted states from OIDC/OAuth2 along with Setting Scopes (go-gitea#16766)
  * Do not use thin scrollbars on Firefox (go-gitea#16738)
  * Download LFS in git and web workflow from minio/s3 directly (SERVE_DIRECT) (go-gitea#16731)
  * Compute proper foreground color for labels (go-gitea#16729)
  * Add edit button to wiki sidebar and footer (go-gitea#16719)
  * Fix migration svg color (go-gitea#16715)
  * Add link to vscode to repo header (go-gitea#16664)
  * Add filter by owner and team to issue/pulls search endpoint (go-gitea#16662)
  * Kanban colored boards (go-gitea#16647)
  * Allow setting X-FRAME-OPTIONS (go-gitea#16643)
  * Separate open and closed issue in metrics (go-gitea#16637)
  * Support direct comparison (git diff a..b) as well merge comparison (a…b) (go-gitea#16635)
  * Add setting to OAuth handlers to skip local 2FA authentication (go-gitea#16594)
  * Make PR merge options more intuitive (go-gitea#16582)
  * Show correct text when comparing commits on empty pull request (go-gitea#16569)
  * Pre-fill suggested New File 'name' and 'content' with Query Params (go-gitea#16556)
  * Add an abstract json layout to make it's easier to change json library (go-gitea#16528)
  * Make Mermaid.js limit configurable (go-gitea#16519)
  * Improve 2FA autofill (go-gitea#16473)
  * Add modals to Organization and Team remove/leave (go-gitea#16471)
  * Show tag name on dashboard items list (go-gitea#16466)
  * Change default cron schedules from @every 24h to @midnight (go-gitea#16431)
  * Prevent double sanitize (go-gitea#16386)
  * Replace `list.List` with slices (go-gitea#16311)
  * Add configuration option to restrict users by default (go-gitea#16256)
  * Move login out of models (go-gitea#16199)
  * Support pagination of organizations on user settings pages (go-gitea#16083)
  * Switch migration icon to svg (go-gitea#15954)
  * Add left padding for chunk header of split diff view (go-gitea#13397)
  * Allow U2F 2FA without TOTP (go-gitea#11573)
* BUGFIXES
  * GitLab reviews may not have the updated_at field set (go-gitea#18450) (go-gitea#18461)
  * Fix detection of no commits when the default branch is not master (go-gitea#18422) (go-gitea#18423)
  * Fix broken oauth2 authentication source edit page (go-gitea#18412) (go-gitea#18419)
  * Place inline diff comment dialogs on split diff in 4th and 8th columns (go-gitea#18403) (go-gitea#18404)
  * Fix restore without topic failure (go-gitea#18387) (go-gitea#18400)
  * Fix commit's time (go-gitea#18375) (go-gitea#18392)
  * Fix partial cloning a repo (go-gitea#18373) (go-gitea#18377)
  * Stop trimming preceding and suffixing spaces from editor filenames (go-gitea#18334)
  * Prevent showing webauthn error for every time visiting `/user/settings/security` (go-gitea#18386)
  * Fix mime-type detection for HTTP server (go-gitea#18370) (go-gitea#18371)
  * Stop trimming preceding and suffixing spaces from editor filenames (go-gitea#18334)
  * Restore propagation of ErrDependenciesLeft (go-gitea#18325)
  * Fix PR comments UI (go-gitea#18323)
  * Use indirect comparison when showing pull requests (go-gitea#18313)
  * Replace satori/go.uuid with gofrs/uuid (go-gitea#18311)
  * Fix commit links on compare page (go-gitea#18310)
  * Don't show double error response in git hook (go-gitea#18292)
  * Handle missing default branch better in owner/repo/branches page (go-gitea#18290)
  * Fix CheckRepoStats and reuse it during migration (go-gitea#18264)
  * Prevent underline hover on cards (go-gitea#18259)
  * Don't delete branch if other PRs with this branch are open (go-gitea#18164)
  * Require codereview to have content (go-gitea#18156)
  * Allow admin to associate missing LFS objects for repositories (go-gitea#18143)
  * When attempting to subscribe other user to issue report why access denied (go-gitea#18091)
  * Add option to convert CRLF to LF line endings for sendmail (go-gitea#18075)
  * Only create pprof files for gitea serv if explicitly asked for (go-gitea#18068)
  * Abort merge if head has been updated before pressing merge (go-gitea#18032)
  * Improve TestPatch to use git read-tree -m and implement git-merge-one-file functionality (go-gitea#18004)
  * Use JSON module instead of stdlib json (go-gitea#18003)
  * Fixed issue merged/closed wording (go-gitea#17973)
  * Return nicer error for ForcePrivate (go-gitea#17971)
  * Fix overflow in commit graph (go-gitea#17947)
  * Prevent services/mailer/mailer_test.go tests from deleteing data directory (go-gitea#17941)
  * Use disable_form_autofill on Codebase and Gitbucket (go-gitea#17936)
  * Fix a panic in NotifyCreateIssueComment (caused by string truncation) (go-gitea#17928)
  * Fix markdown URL parsing (go-gitea#17924)
  * Apply CSS Variables to all message elements (go-gitea#17920)
  * Improve checkBranchName (go-gitea#17901)
  * Update chi/middleware to chi/v5/middleware (go-gitea#17888)
  * Fix position of label color picker colors (go-gitea#17866)
  * Fix ListUnadoptedRepositories incorrect total count (go-gitea#17865)
  * Remove whitespace inside rendered code `<td>` (go-gitea#17859)
  * Make Co-committed-by and co-authored-by trailers optional (go-gitea#17848)
  * Fix value of User.IsRestricted when oauth2 user registration (go-gitea#17839)
  * Use new OneDev /milestones endpoint (go-gitea#17782)
  * Prevent deadlock in TestPersistableChannelQueue (go-gitea#17717)
  * Simplify code for writing SHA to name-rev (go-gitea#17696)
  * Fix database deadlock when update issue labels (go-gitea#17649)
  * Add warning for BIDI characters in page renders and in diffs (go-gitea#17562)
  * Fix ipv6 parsing for builtin ssh server (go-gitea#17561)
  * Multiple Escaping Improvements (go-gitea#17551)
  * Fixes go-gitea#16559 - Do not trim leading spaces for tab delimited (go-gitea#17442)
  * Show client-side error if wiki page is empty (go-gitea#17415)
  * Fix context popup error (go-gitea#17398)
  * Stop sanitizing full name in API (go-gitea#17396)
  * Fix issue close/comment buttons on mobile (go-gitea#17317)
  * Fix navbar UI (go-gitea#17235)
  * Fix problem when database id is not increment as expected (go-gitea#17229)
  * Open the DingTalk link in browser (go-gitea#17084)
  * Remove heads pointing to missing old refs (go-gitea#17076)
  * Fix commit status index problem (go-gitea#17061)
  * Handle broken references in mirror sync (go-gitea#17013)
  * Fix for create repo page layout (go-gitea#17012)
  * Improve LDAP synchronization efficiency (go-gitea#16994)
  * Add repo_id for attachment (go-gitea#16958)
  * Clean-up HookPreReceive and restore functionality for pushing non-standard refs (go-gitea#16705)
  * Remove duplicate csv import in modules/csv/csv.go (go-gitea#16631)
  * Improve SMTP authentication and Fix user creation bugs  (go-gitea#16612)
  * Fixed emoji alias not parsed in links (go-gitea#16221)
  * Calculate label URL on API  (go-gitea#16186)
* TRANSLATION
  * Fix mispelling of starred as stared (go-gitea#17465)
  * Re-separate the color translation strings (go-gitea#17390)
  * Enable Malayalam, Greek, Persian, Hungarian & Indonesian by default (go-gitea#16998)
* BUILD
  * Add lockfile-check (go-gitea#18285)
  * Don't store assets modified time into generated files (go-gitea#18193)
  * Use shadowing script for docker (go-gitea#17846)
* MISC
  * Update JS dependencies (go-gitea#17611)

Signed-off-by: Andrew Thornton <art27@cantab.net>
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/enhancement An improvement of existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CSV Renderer may guess a TSV file is comma delimited even when tab delimited
8 participants