embedded subcommand not working in Linux #12082

SuperSandro2000 · 2020-06-28T13:22:10Z

Gitea version (or commit ref): 1.12.1 (till latest)

Description

The embedded subcommand does not work list or extract any files.

Screenshots

bash-5.0# gitea -v
Gitea version 1.12.1 built with GNU Make 4.2.1, go1.14.4 : bindata, sqlite, sqlite_unlock_notify
bash-5.0# gitea embedded list '**'
bash-5.0#

Reason

Gitea embedded can not run with mismatched Linux user in an installed config, there will be no output at all.

Workaround

Run gitea with the correct Linux user (as in the app.ini), using su or sudo

The text was updated successfully, but these errors were encountered:

lunny · 2020-07-05T04:22:56Z

~~In fact the command name is embed not embedded~~

SuperSandro2000 · 2020-07-06T14:03:26Z

Yeah, this is wrong in the help:

bash-5.0# gitea help                                                                                                                                                                                              
NAME:                                                                                                                                                                                                             
   Gitea - A painless self-hosted Git service                                                                                                                                                                     
                                                                                                                                                                                                                  
USAGE:                                                                                                                                                                                                            
   gitea [global options] command [command options] [arguments...]                                                                                                                                                
                                                                                                                                                                                                                  
VERSION:                                                                                                                                                                                                          
   1.12.1 built with GNU Make 4.2.1, go1.14.4 : bindata, sqlite, sqlite_unlock_notify                                                                                                                             
                                                                                                                                                                                                                  
DESCRIPTION:                                                                                                                                                                                                      
   By default, gitea will start serving using the webserver with no                                                                                                                                               
arguments - which can alternatively be run by running the subcommand web.                                                                                                                                         
                                                                                                                                                                                                                  
COMMANDS:                                                                                                                                                                                                         
     web       Start Gitea web server                                                                                                                                                                             
     serv      This command should only be called by SSH shell                                                                                                                                                    
     hook      Delegate commands to corresponding Git hooks                                                                                                                                                       
     dump      Dump Gitea files and database                                                                                                                                                                      
     cert      Generate self-signed certificate                                                                                                                                                                   
     admin     Command line interface to perform common administrative operations                                                                                                                                 
     generate  Command line interface for running generators                                                                                                                                                      
     migrate   Migrate the database                                                                                                                                                                               
     keys      This command queries the Gitea database to get the authorized command for a given ssh key fingerprint                                                                                              
     convert   Convert the database                                                                                                                                                                               
     doctor    Diagnose problems                                                                                                                                                                                  
     manager   Manage the running gitea process                                                                                                                                                                   
     embedded  Extract embedded resources                                                                                                                                                                         
     help, h   Shows a list of commands or help for one command                                                                                                                                                   
                                                                                                                                                                                                                  
GLOBAL OPTIONS:                                                                                                                                                                                                   
   --port value, -p value         Temporary port number to prevent conflict (default: "3000")                                                                                                                     
   --pid value, -P value          Custom pid file path (default: "/var/run/gitea.pid")                                                                                                                            
   --custom-path value, -C value  Custom path file path (default: "/data/gitea")                                                                                                                                  
   --config value, -c value       Custom configuration file path (default: "/data/gitea/conf/app.ini")                                                                                                            
   --version, -v                  print the version                                                                                                                                                               
   --work-path value, -w value    Set the gitea working path (default: "/usr/local/bin")                                                                                                                          
   --help, -h                     show help     
                                                                                                                                                                                                                  
GLOBAL OPTIONS:                                                                                                                                                                                                   
   --port value, -p value         Temporary port number to prevent conflict (default: "3000")                                                                                                                     
   --pid value, -P value          Custom pid file path (default: "/var/run/gitea.pid")                                                                                                                            
   --custom-path value, -C value  Custom path file path (default: "/data/gitea")                                                                                                                                  
   --config value, -c value       Custom configuration file path (default: "/data/gitea/conf/app.ini")                                                                                                            
   --version, -v                  print the version                                                                                                                                                               
   --work-path value, -w value    Set the gitea working path (default: "/usr/local/bin")                                                                                                                          
   --help, -h                     show help                                                                                                                                                                       
                                                                                                                                                                                                                  
DEFAULT CONFIGURATION:                                                                                                                                                                                            
     CustomPath:  /data/gitea (GITEA_CUSTOM)                                                                                                                                                                      
     CustomConf:  /data/gitea/conf/app.ini                                                                                                                                                                        
     AppPath:     /usr/local/bin/gitea                                                                                                                                                                            
     AppWorkPath: /usr/local/bin

ghost · 2020-07-29T20:00:49Z

I'm confused by this response. embedded seems to be parsed properly while embed (or any other random string) merely attempts to boot up gitea.

SuperSandro2000 · 2020-07-30T17:55:57Z

If the command is not understood it should error.

jolheiser · 2020-07-30T17:59:26Z

embedded is the correct sub-command name.
The reason strings that aren't valid sub-commands boot up gitea is because of

gitea/main.go

Lines 98 to 101 in 838fef2

    
           // Set the default to be equivalent to cmdWeb and add the default flags 
        
           app.Flags = append(app.Flags, cmd.CmdWeb.Flags...) 
        
           app.Flags = append(app.Flags, defaultFlags...) 
        
           app.Action = cmd.CmdWeb.Action

We are setting the default command to be the same we gitea web

As for why the command isn't working for you, you need to specify the config file with the -c flag.

e.g. gitea -c /etc/gitea/app.ini embedded list

ghost · 2020-07-30T20:51:06Z

I'm getting empty file listings in docker exec, while the paths are being detected without flags:

DEFAULT CONFIGURATION:
     CustomPath:  /data/gitea (GITEA_CUSTOM)
     CustomConf:  /data/gitea/conf/app.ini
     AppPath:     /usr/local/bin/gitea
     AppWorkPath: /usr/local/bin

I've however tried adding the flag, and no files were listed still.

/ # gitea -c /data/gitea/conf/app.ini embedded list '**'
/ #

stale · 2020-09-29T05:34:52Z

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

SuperSandro2000 · 2020-09-29T08:00:23Z

bump

zeripath · 2020-09-29T21:07:59Z

I can't replicate your problem.

This works on gitea/gitea:latest:

docker pull gitea/gitea:latest
docker run gitea/gitea:latest &
docker exec $(docker ps | grep gitea | cut -f1 -d\ ) gitea embedded list "**"

This works on gitea/gitea:1.12:

docker pull gitea/gitea:1.12
docker run gitea/gitea:1.12 &
docker exec $(docker ps | grep gitea | cut -f1 -d\ ) gitea embedded list "**"

view appears to work

docker exec $(docker ps | grep gitea | cut -f1 -d\ ) gitea embedded view templates/user/settings/keys.tmpl

extract also works.

docker exec $(docker ps | grep gitea | cut -f1 -d\ ) gitea embedded extract templates/user/auth/change_passwd_inner.tmpl
docker exec $(docker ps | grep gitea | cut -f1 -d\ ) cat templates/user/auth/change_passwd_inner.tmpl

Too many docker users are caught out by the default location for the app.ini file being environment dependent so that when they docker exec into the container the `gitea` commands do not work properly and require additional `-c` arguments to correctly pick up the configuration. This PR simply builds the default locations into the build. Fix go-gitea#14468 Reference go-gitea#17497 Reference go-gitea#12082 Reference go-gitea#8941 ... amongst others ... Signed-off-by: Andrew Thornton <art27@cantab.net>

rmawatson · 2021-11-27T15:49:35Z

This issue still seems to be present. I can get a result from embedded list but only by specifying an invalid config file.

(1) With invalid config file - works

bash-5.1# gitea -c garbage/app.ini embedded list templates/user/settings/keys.tmpl
2021/11/27 13:46:59 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/garbage/app.ini'
2021/11/27 13:46:59 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/garbage/app.ini'
2021/11/27 13:46:59 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/garbage/app.ini'
templates/user/settings/keys.tmpl

(2) With correct path to config file - No Result

bash-5.1# gitea -c conf/app.ini embedded list templates/user/settings/keys.tmpl
2021/11/27 13:47:46 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/conf/app.ini'
2021/11/27 13:47:46 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/conf/app.ini'
2021/11/27 13:47:46 ...s/setting/setting.go:547:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/conf/app.ini'
bash-5.1#

(3) With no path specified (presumably using /data/gitea/conf/app.ini) - No Result

bash-5.1# gitea embedded list templates/user/settings/keys.tmpl
bash-5.1#

stat showing the config file exists at the location used for (2) (3)

bash-5.1# stat /data/gitea/conf/app.ini
  File: /data/gitea/conf/app.ini
  Size: 2449            Blocks: 8          IO Block: 4096   regular file
Device: 802h/2050d      Inode: 31590370    Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/     git)   Gid: ( 1000/     git)
Access: 2021-11-27 04:13:00.305459938 +0100
Modify: 2021-11-27 10:49:48.344909082 +0100
Change: 2021-11-27 10:49:48.344909082 +0100

So a workaround seems to be to specify an incorrect config file. I'm using gitea docker 1.15.6

wxiaoguang · 2021-11-27T16:01:26Z

I meet the similar problem on my side: embedded list outputs nothing.

If I do TAGS=bindata make build (1.16) on macOS, then ./gitea embedded list can output the correct list.

But if I do GOOS=... TAGS=bindata make build (1.16) and copy the binary into the Linux docker, ./gitea embedded list outputs nothing. (I know my operation is a little strange, but whatever happens the output should not be empty)

wxiaoguang · 2021-11-27T17:45:32Z

Some thoughts (maybe conclusions):

This bug is not Docker-related: I can reproduce it in a real Linux server
This bug is not config (app.ini) related: The embedded sub-commands don't need it
The embedded list and embedded view may output nothing (even with the with non-existing file paths): it's abnormal, embedded view should output error messages if the target doesn't exist.
The bug seems related with the console output mechanism (according 3)

ps: I haven't investigated this problem carefully, these information is just FYI, correct me if I was wrong.

Too many docker users are caught out by the default location for the app.ini file being environment dependent so that when they docker exec into the container the gitea commands do not work properly and require additional -c arguments to correctly pick up the configuration. This PR simply shadows the gitea binary using variants of the FHS compatible script to make the command gitea have the default locations by default. Fix go-gitea#14468 Reference go-gitea#17497 Reference go-gitea#12082 Reference go-gitea#8941 ... amongst others ... Replace go-gitea#17501 Signed-off-by: Andrew Thornton <art27@cantab.net>

Too many docker users are caught out by the default location for the app.ini file being environment dependent so that when they docker exec into the container the gitea commands do not work properly and require additional -c arguments to correctly pick up the configuration. This PR simply shadows the gitea binary using variants of the FHS compatible script to make the command gitea have the default locations by default. Fix #14468 Reference #17497 Reference #12082 Reference #8941 ... amongst others ... Replace #17501 Signed-off-by: Andrew Thornton <art27@cantab.net>

wxiaoguang · 2021-12-06T02:56:35Z

@lunny I think this is a bug, not related to docker. See my comments above. So maybe we should keep this open.

wxiaoguang · 2021-12-29T09:07:58Z

I found the root case of this bug.

There is a check:

gitea/modules/setting/setting.go

Line 954 in 72f9050

log.Fatal("Expect user '%s' but current user is: %s", RunUser, currentUser)

. The code is pretty old (as early as 2014)
The embedded command removes loggers:

gitea/cmd/embedded.go

Line 115 in 72f9050

log.DelNamedLogger(log.DEFAULT)

Then bad thing happens: Gitea embedded can not run with mismatched Linux user in an installed config, and no error is outputted.

Too many docker users are caught out by the default location for the app.ini file being environment dependent so that when they docker exec into the container the gitea commands do not work properly and require additional -c arguments to correctly pick up the configuration. This PR simply shadows the gitea binary using variants of the FHS compatible script to make the command gitea have the default locations by default. Fix go-gitea#14468 Reference go-gitea#17497 Reference go-gitea#12082 Reference go-gitea#8941 ... amongst others ... Replace go-gitea#17501 Signed-off-by: Andrew Thornton <art27@cantab.net>

grubjack · 2023-01-17T10:25:30Z

working that way:

# ls -l conf/app.ini
-rw-r--r-- 1 git git 2611 Jan 16 19:09 conf/app.ini

# su git -s /usr/local/bin/gitea embedded list

## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>

lunny added the type/docs This PR mainly updates/creates documentation label Jul 5, 2020

stale bot added the issue/stale label Sep 29, 2020

stale bot removed the issue/stale label Sep 29, 2020

zeripath mentioned this issue Oct 31, 2021

Use LDFLAGS to set the default locations in docker #17501

Closed

wxiaoguang added type/bug and removed type/docs This PR mainly updates/creates documentation labels Nov 27, 2021

This comment was marked as off-topic.

Sign in to view

This comment has been minimized.

Sign in to view

zeripath mentioned this issue Nov 28, 2021

Use shadowing script for docker #17846

Merged

lunny closed this as completed Dec 6, 2021

wxiaoguang reopened this Dec 6, 2021

wxiaoguang changed the title ~~embedded subcommand not working in Docker~~ embedded subcommand not working in Linux Dec 6, 2021

wxiaoguang added the issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented label Dec 29, 2021

wxiaoguang mentioned this issue Dec 29, 2021

(Discontinued) Show error logs in embedded sub-command #18121

Closed

wxiaoguang linked a pull request Dec 29, 2021 that will close this issue

(Discontinued) Show error logs in embedded sub-command #18121

Closed

wxiaoguang added the issue/workaround it is or has a workaround label Jun 25, 2022

wxiaoguang mentioned this issue Jul 23, 2022

Refactor legacy git init #20376

Merged

lunny mentioned this issue Apr 27, 2023

Merge setting.InitXXX into one function with options #24389

Merged

wxiaoguang mentioned this issue May 15, 2023

Rewrite logger system #24726

Merged

3 tasks

silverwind closed this as completed in #24726 May 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jul 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

embedded subcommand not working in Linux #12082

embedded subcommand not working in Linux #12082

SuperSandro2000 commented Jun 28, 2020 •

edited by wxiaoguang

lunny commented Jul 5, 2020 •

edited

SuperSandro2000 commented Jul 6, 2020

ghost commented Jul 29, 2020

SuperSandro2000 commented Jul 30, 2020

jolheiser commented Jul 30, 2020 •

edited

ghost commented Jul 30, 2020

stale bot commented Sep 29, 2020

SuperSandro2000 commented Sep 29, 2020

zeripath commented Sep 29, 2020

rmawatson commented Nov 27, 2021 •

edited

wxiaoguang commented Nov 27, 2021 •

edited

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment has been minimized.

This comment has been minimized.

wxiaoguang commented Nov 27, 2021

wxiaoguang commented Dec 6, 2021

wxiaoguang commented Dec 29, 2021 •

edited

grubjack commented Jan 17, 2023 •

edited

embedded subcommand not working in Linux #12082

embedded subcommand not working in Linux #12082

Comments

SuperSandro2000 commented Jun 28, 2020 • edited by wxiaoguang

Description

Screenshots

Reason

Workaround

lunny commented Jul 5, 2020 • edited

SuperSandro2000 commented Jul 6, 2020

ghost commented Jul 29, 2020

SuperSandro2000 commented Jul 30, 2020

jolheiser commented Jul 30, 2020 • edited

ghost commented Jul 30, 2020

stale bot commented Sep 29, 2020

SuperSandro2000 commented Sep 29, 2020

zeripath commented Sep 29, 2020

rmawatson commented Nov 27, 2021 • edited

wxiaoguang commented Nov 27, 2021 • edited

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment has been minimized.

This comment has been minimized.

wxiaoguang commented Nov 27, 2021

wxiaoguang commented Dec 6, 2021

wxiaoguang commented Dec 29, 2021 • edited

grubjack commented Jan 17, 2023 • edited

SuperSandro2000 commented Jun 28, 2020 •

edited by wxiaoguang

lunny commented Jul 5, 2020 •

edited

jolheiser commented Jul 30, 2020 •

edited

rmawatson commented Nov 27, 2021 •

edited

wxiaoguang commented Nov 27, 2021 •

edited

wxiaoguang commented Dec 29, 2021 •

edited

grubjack commented Jan 17, 2023 •

edited