Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chalk marks and reports may lack git keys when data is packed #342

Closed
ee7 opened this issue Jun 14, 2024 · 3 comments · Fixed by #341
Closed

chalk marks and reports may lack git keys when data is packed #342

ee7 opened this issue Jun 14, 2024 · 3 comments · Fixed by #341

Comments

@ee7
Copy link
Contributor

ee7 commented Jun 14, 2024 •
edited
Loading

Example

#!/bin/sh
set +e

tmpdir='/tmp/chalk-git-pack'
rm -rf "${tmpdir}"
mkdir -p "${tmpdir}"
cd "${tmpdir}" || exit

git init
git config user.email "foo@example.com" # In case the user deliberately doesn't set these globally.
git config user.name "Foo Bar"
cp /path/to/chalk ./chalk # To be explicit about the chalk version used.

echo 'hi' > hi.txt
git add hi.txt
git commit -m 'initial commit'
./chalk insert hi.txt
git gc
./chalk insert hi.txt

Expected behavior

The chalk report from each chalk insert command contains git key/value pairs, like:

    "_AUTHOR": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_BRANCH": "main",
    "_COMMITTER": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_COMMIT_ID": "854fe875bc3a2e11d55bc829a592aef8dbe189b9",
    "_COMMIT_MESSAGE": "initial commit",
    "_COMMIT_SIGNED": false,
    "_DATE_AUTHORED": "Fri Jun 14 14:29:38 2024 +0200",
    "_DATE_COMMITTED": "Fri Jun 14 14:29:38 2024 +0200",

Observed behavior

The output after git gc may lack those keys, which points to a general issue. And we see the warning:

warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.

Diagnosis

Chalk should handle packfiles better.

Related: #332
@ee7 ee7 mentioned this issue Jun 14, 2024
Closed
@miki725
Copy link
Contributor

miki725 commented Jun 14, 2024

whats the full output with --trace?

@ee7
Copy link
Contributor Author

ee7 commented Jun 14, 2024

Yeah, I should've mentioned that - sorry. The main thing is the warning:

warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.

but for the first chalk insert (using a chalk built from either main, or 53b7086 on PR #341):

trace: Loaded sink config for 'default_out'
trace: Loaded sink config for 'console_out'
trace: Loaded sink config for 'virtual_chalk_log'
trace: Installed plugin: docker
trace: Installed plugin: elf
trace: Installed plugin: macos
trace: Installed plugin: python_pyc
trace: Installed plugin: source
trace: Installed plugin: zip
trace: Installed plugin: zippeditem
trace: Installed plugin: elf_last_resort
trace: Installed plugin: ci_github
trace: Installed plugin: ci_gitlab
trace: Installed plugin: ci_jenkins
trace: Installed plugin: conffile
trace: Installed plugin: aws_ecs
trace: Installed plugin: aws_lambda
trace: Installed plugin: tool
trace: Installed plugin: cloud_metadata
trace: Installed plugin: authors
trace: Installed plugin: github_codeowners
trace: Installed plugin: procfs
trace: Installed plugin: system
trace: Installed plugin: metsys
trace: Installed plugin: vctl_git
trace: Installed plugin: tech_stack_generic
trace: Checking chalk binary '/tmp/chalk-git-pack/chalk' for embedded config
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/chalk
trace: /tmp/chalk-git-pack/chalk: scanning file
trace: /tmp/chalk-git-pack/chalk: chalk mark magic @ 10874261
trace: No embedded chalk mark.
trace: Using the default user config.  See 'chalk dump' to view.
trace: Looking for config file at: /etc/chalk/chalk.c4m
trace: /etc/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /etc/chalk.c4m
trace: /etc/chalk.c4m: No configuration file found.
trace: Looking for config file at: /tmp/chalk-git-pack/chalk.c4m
trace: /tmp/chalk-git-pack/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/.config/chalk/chalk.c4m
trace: /home/foo/.config/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/chalk.c4m
trace: /home/foo/chalk.c4m: No configuration file found.
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/home/foo/.local/share/bash_completion/completions/chalk.bash"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
info:  Invalid chalk mark in autocompletion script. Updating.
info:  Installed bash auto-completion file to: /home/foo/.local/share/bash_completion/completions/chalk.bash
trace: Collecting host-level chalk-time data
trace: Collecting chalk time artifact info
trace: Running plugin: system
trace: Running plugin: ci_jenkins
trace: Running plugin: ci_gitlab
trace: Running plugin: tool
trace: Running plugin: ci_github
trace: Running plugin: aws_ecs
trace: ecs: metadata env var is not defined: no AWS info available
trace: Running plugin: conffile
trace: Called artifacts() -- filepaths = @["/tmp/chalk-git-pack/hi.txt"]; otherPaths = @[]
trace: Asking codec 'python_pyc' to scan artifacts.
trace: Codec python_pyc: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: /tmp/chalk-git-pack/hi.txt: ignored due to matching ignore pattern: .*\.txt
trace: We will NOT report additional path skips.
trace: Asking codec 'zip' to scan artifacts.
trace: Codec zip: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'macos' to scan artifacts.
trace: Codec macos: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf' to scan artifacts.
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'source' to scan artifacts.
trace: Codec source: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf_last_resort' to scan artifacts.
trace: Codec elf_last_resort: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Collecting runtime host info.
trace: Collecting run time host info
trace: Running plugin: system
trace: Running plugin: docker
trace: Searching PATH for docker
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/usr/bin/docker"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
trace: Found 'docker' in PATH: /usr/bin/docker
trace: docker: client version: 26.1.4
trace: docker: server version: 26.1.4
trace: docker: buildx version: 0.15.0
trace: Running plugin: tech_stack_generic
trace: Skipping tech stack runtime detection plugin
trace: Running plugin: vctl_git
trace: Looking for .git directory, from: /tmp/chalk-git-pack/hi.txt
trace: Found version control dir: /tmp/chalk-git-pack/.git
trace: branch: main
trace: commit ID: 854fe875bc3a2e11d55bc829a592aef8dbe189b9
trace: Running plugin: cloud_metadata
trace: Not an EC2 instance - skipping check for IMDSv2
trace: Running plugin: aws_ecs
trace: Running plugin: aws_lambda
trace: lambda: function env vars are not defined: no AWS info available
trace: Running plugin: conffile
trace: Running plugin: metsys
trace: Generating command report.
trace: /tmp/chalk-git-pack/chalk-reports.jsonl: file lock obtained, but no report cache to read.
info:  /home/foo/.local/chalk/chalk.log: Open (sink conf='default_out')
	filename        = "/home/foo/.local/chalk/chalk.log"
	log_search_path = ["/var/log/chalk/", "~/.log/chalk/", "."]
info:  Full chalk report appended to: ~/.local/chalk/chalk.log
trace: Published the report for topic 'report' (1 subscribers)
trace: Loaded sink config for 'json_console_out'
[
  {
    "_OPERATION": "insert",
    "_DATETIME": "2024-06-14T14:29:38.647+02:00",
    "_AUTHOR": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_BRANCH": "main",
    "_COMMITTER": "Foo Bar <foo@example.com> 1718368178 +0200",
    "_COMMIT_ID": "854fe875bc3a2e11d55bc829a592aef8dbe189b9",
    "_COMMIT_MESSAGE": "initial commit",
    "_COMMIT_SIGNED": false,
    "_DATE_AUTHORED": "Fri Jun 14 14:29:38 2024 +0200",
    "_DATE_COMMITTED": "Fri Jun 14 14:29:38 2024 +0200",
    "_ENV": {
      "PWD": "/tmp/chalk-git-pack",
      "XDG_SESSION_TYPE": "wayland",
      "USER": "foo",
      "PATH": "/usr/local/sbin:/usr/local/bin:/usr/bin"
    },
    "_OP_ARGV": [
      "/tmp/chalk-git-pack/chalk",
      "insert",
      "--trace",
      "hi.txt"
    ],
    "_OP_CHALKER_VERSION": "0.4.5-dev",
    "_ORIGIN_URI": "local",
    "_OP_CHALK_COUNT": 0,
    "_OP_UNMARKED_COUNT": 0
  }
]
trace: Published the report for topic 'terminal_chalk_time' (1 subscribers)

and the second (after the git gc):

trace: Loaded sink config for 'default_out'
trace: Loaded sink config for 'console_out'
trace: Loaded sink config for 'virtual_chalk_log'
trace: Installed plugin: docker
trace: Installed plugin: elf
trace: Installed plugin: macos
trace: Installed plugin: python_pyc
trace: Installed plugin: source
trace: Installed plugin: zip
trace: Installed plugin: zippeditem
trace: Installed plugin: elf_last_resort
trace: Installed plugin: ci_github
trace: Installed plugin: ci_gitlab
trace: Installed plugin: ci_jenkins
trace: Installed plugin: conffile
trace: Installed plugin: aws_ecs
trace: Installed plugin: aws_lambda
trace: Installed plugin: tool
trace: Installed plugin: cloud_metadata
trace: Installed plugin: authors
trace: Installed plugin: github_codeowners
trace: Installed plugin: procfs
trace: Installed plugin: system
trace: Installed plugin: metsys
trace: Installed plugin: vctl_git
trace: Installed plugin: tech_stack_generic
trace: Checking chalk binary '/tmp/chalk-git-pack/chalk' for embedded config
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/chalk
trace: /tmp/chalk-git-pack/chalk: scanning file
trace: /tmp/chalk-git-pack/chalk: chalk mark magic @ 10874261
trace: No embedded chalk mark.
trace: Using the default user config.  See 'chalk dump' to view.
trace: Looking for config file at: /etc/chalk/chalk.c4m
trace: /etc/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /etc/chalk.c4m
trace: /etc/chalk.c4m: No configuration file found.
trace: Looking for config file at: /tmp/chalk-git-pack/chalk.c4m
trace: /tmp/chalk-git-pack/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/.config/chalk/chalk.c4m
trace: /home/foo/.config/chalk/chalk.c4m: No configuration file found.
trace: Looking for config file at: /home/foo/chalk.c4m
trace: /home/foo/chalk.c4m: No configuration file found.
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/home/foo/.local/share/bash_completion/completions/chalk.bash"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
info:  Invalid chalk mark in autocompletion script. Updating.
info:  Installed bash auto-completion file to: /home/foo/.local/share/bash_completion/completions/chalk.bash
trace: Collecting host-level chalk-time data
trace: Collecting chalk time artifact info
trace: Running plugin: system
trace: Running plugin: ci_jenkins
trace: Running plugin: ci_gitlab
trace: Running plugin: tool
trace: Running plugin: ci_github
trace: Running plugin: aws_ecs
trace: ecs: metadata env var is not defined: no AWS info available
trace: Running plugin: conffile
trace: Called artifacts() -- filepaths = @["/tmp/chalk-git-pack/hi.txt"]; otherPaths = @[]
trace: Asking codec 'python_pyc' to scan artifacts.
trace: Codec python_pyc: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: /tmp/chalk-git-pack/hi.txt: ignored due to matching ignore pattern: .*\.txt
trace: We will NOT report additional path skips.
trace: Asking codec 'zip' to scan artifacts.
trace: Codec zip: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'macos' to scan artifacts.
trace: Codec macos: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf' to scan artifacts.
trace: Codec elf: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'source' to scan artifacts.
trace: Codec source: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Asking codec 'elf_last_resort' to scan artifacts.
trace: Codec elf_last_resort: beginning scan of /tmp/chalk-git-pack/hi.txt
trace: Collecting runtime host info.
trace: Collecting run time host info
trace: Running plugin: system
trace: Running plugin: docker
trace: Searching PATH for docker
trace: Running subscan. Command name is temporarily: extract
trace: Subscan location: @["/usr/bin/docker"]
trace: *** Setting log-level = "error" for scan.  Use --debug to turn on
trace: Subscan done. Restored command name to: insert
trace: Found 'docker' in PATH: /usr/bin/docker
trace: docker: client version: 26.1.4
trace: docker: server version: 26.1.4
trace: docker: buildx version: 0.15.0
trace: Running plugin: tech_stack_generic
trace: Skipping tech stack runtime detection plugin
trace: Running plugin: vctl_git
trace: Looking for .git directory, from: /tmp/chalk-git-pack/hi.txt
trace: Found version control dir: /tmp/chalk-git-pack/.git
trace: branch: main
warn:  /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.
trace: Running plugin: cloud_metadata
trace: Not an EC2 instance - skipping check for IMDSv2
trace: Running plugin: aws_ecs
trace: Running plugin: aws_lambda
trace: lambda: function env vars are not defined: no AWS info available
trace: Running plugin: conffile
trace: Running plugin: metsys
trace: Generating command report.
trace: /tmp/chalk-git-pack/chalk-reports.jsonl: file lock obtained, but no report cache to read.
info:  /home/foo/.local/chalk/chalk.log: Open (sink conf='default_out')
	filename        = "/home/foo/.local/chalk/chalk.log"
	log_search_path = ["/var/log/chalk/", "~/.log/chalk/", "."]
info:  Full chalk report appended to: ~/.local/chalk/chalk.log
trace: Published the report for topic 'report' (1 subscribers)
trace: Loaded sink config for 'json_console_out'
[
  {
    "_OPERATION": "insert",
    "_DATETIME": "2024-06-14T14:29:40.704+02:00",
    "_ENV": {
      "PWD": "/tmp/chalk-git-pack",
      "XDG_SESSION_TYPE": "wayland",
      "USER": "foo",
      "PATH": "/usr/local/sbin:/usr/local/bin:/usr/bin"
    },
    "_OP_ARGV": [
      "/tmp/chalk-git-pack/chalk",
      "insert",
      "--trace",
      "hi.txt"
    ],
    "_OP_CHALKER_VERSION": "0.4.5-dev",
    "_OP_CHALK_COUNT": 0,
    "_OP_UNMARKED_COUNT": 0
  }
]
trace: Published the report for topic 'terminal_chalk_time' (1 subscribers)

@miki725 miki725 mentioned this issue Jun 14, 2024
Merged
4 tasks
@miki725
Copy link
Contributor

miki725 commented Jun 14, 2024

yeah git packs refs themeselves as well

warn: /tmp/chalk-git-pack/.git/refs/heads/main: Git ref file for 'main' doesnt exist. Most likely it's an empty git repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(git): git packed objects/refs crashappsec/chalk
2 participants
@miki725 @ee7