Problems to open checkpoint with journald

[Pedneri1]

Vector Version

vector 0.10.0 (g0f0311a x86_64-unknown-linux-gnu 2020-07-22)

Vector Configuration File

[sources.in]
  type = "journald" # required

[transforms.fields_filter]
  type = "remove_fields" # required
  inputs = ["in"] #required
  fields = ["_MACHINE_ID", "_SELINUX_CONTEXT", "_SYSTEMD_INVOCATION_ID", "_SYSTEMD_SLICE", "_TRANSPORT", "_CMDLINE", "_CAP_EFFECTIVE", "_COMM", "_GID", "_UID", "source_type", "_BOOT_ID"]

[sinks.out]
  # Encoding
  encoding.codec = "json" # required

  # General
  group_name = "my-ec2-machine" # required
  inputs = ["fields_filter"] # required
  region = "sa-east-1" # required, required when endpoint = ""
  stream_name = "{{ host }}" # required
  type = "aws_cloudwatch_logs" # required

Expected Behavior

Vector should be sending the journald logs to cloudwatch

Actual Behavior

After creating the config file, I tried to run Vector using the sudo vector --config vector.toml but it freezes after the Healthcheck: Passed. and even after hours, it does nothing.

Aug 25 12:18:56.785  INFO vector: Log level "info" is enabled.
Aug 25 12:18:56.786  INFO vector: Loading configs. path=["vector.toml"]
Aug 25 12:18:56.789  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 19:45:24 +0000" arch="x86_64"
Aug 25 12:18:56.801  INFO vector::topology: Running healthchecks.
Aug 25 12:18:56.802  INFO vector::topology: Starting source "in"
Aug 25 12:18:56.803  INFO vector::topology: Starting transform "fields_filter"
Aug 25 12:18:56.803  INFO vector::topology: Starting sink "out"
Aug 25 12:18:56.819  INFO source{name=in type=journald}: vector::sources::journald: Starting journald server.
Aug 25 12:18:56.909  INFO vector::topology::builder: Healthcheck: Passed.
^CAug 25 12:19:07.325  INFO vector: Shutting down.

And when I move the config file to /etc/vector/ and try to start it as a systemd service sudo systemd start vector it displays this error:

● vector.service - Vector
   Loaded: loaded (/etc/systemd/system/vector.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-08-25 12:16:15 UTC; 4min 42s ago
     Docs: https://vector.dev
  Process: 6357 ExecStart=/usr/bin/vector (code=exited, status=78)
 Main PID: 6357 (code=exited, status=78)

Aug 25 12:16:14 ip-172-31-16-245 systemd[1]: Started Vector.
Aug 25 12:16:14 ip-172-31-16-245 vector[6357]: Aug 25 12:16:14.901  INFO vector: Log level "info" is enabled.
Aug 25 12:16:14 ip-172-31-16-245 vector[6357]: Aug 25 12:16:14.921  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Aug 25 12:16:14 ip-172-31-16-245 vector[6357]: Aug 25 12:16:14.962  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 19:45:24 +0000" arch="x86_64"
Aug 25 12:16:15 ip-172-31-16-245 vector[6357]: Aug 25 12:16:15.023 ERROR vector::topology: Configuration error: Source "in": Unable to open checkpoint file: Permission denied (os error 13)
Aug 25 12:16:15 ip-172-31-16-245 systemd[1]: vector.service: Main process exited, code=exited, status=78/n/a
Aug 25 12:16:15 ip-172-31-16-245 systemd[1]: vector.service: Failed with result 'exit-code'.

The user vector is already on the systemd-journal group

Additional Context

Vector is running on a ec2 machine

References

• journald source fails to retrieve checkpoint #1081

[bruceg]

For the first case, it looks like it is not receiving logs from journalctl. Try setting up the output as type = "console" and run with --verbose --verbose to see if it is receiving anything from journalctl.

For the second case, the vector user does not have permissions to write to the checkpoint directory, which defaults to /var/lib/vector/in for your config. Check that this directory exists and is writable to the vector user.

[Pedneri1]

Following this matter on the second case, I just checked the /var/lib/vector/in permissions

$ cd /var/lib/vector
$ ls -la

and got the following:

drwxr-xr-x  2 vector vector 4096 Aug 19 13:35 in

Vector has permissions for reading and writing on the in/ folder.

[gtie]

@Pedneri1 , did you add the vector user to the systemd-journal group (by executing usermod -aG systemd-journal vector)? I suspect that the error might be misleading.

TR, I'm experimenting with the same setup (journalctl -> cloudwatch logs) and vector has no problems reading up from journalctl after it is added to the appropriate group

[Pedneri1]

@gtie yes, I did. But tried to run it again anyway.

$ sudo usermod -aG systemd-journal vector

than

$ sudo systemctl restart vector
$ sudo systemctl status vector

but the error persists

● vector.service - Vector
   Loaded: loaded (/etc/systemd/system/vector.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-09-04 12:32:49 UTC; 4s ago
     Docs: https://vector.dev
  Process: 22085 ExecStart=/usr/bin/vector (code=exited, status=78)
 Main PID: 22085 (code=exited, status=78)

Sep 04 12:32:48 ip-172-31-16-245 systemd[1]: Started Vector.
Sep 04 12:32:48 ip-172-31-16-245 vector[22085]: Sep 04 12:32:48.997  INFO vector: Log level "info" is enabled.
Sep 04 12:32:49 ip-172-31-16-245 vector[22085]: Sep 04 12:32:49.016  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Sep 04 12:32:49 ip-172-31-16-245 vector[22085]: Sep 04 12:32:49.056  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 19:45:24 +00
Sep 04 12:32:49 ip-172-31-16-245 vector[22085]: Sep 04 12:32:49.122 ERROR vector::topology: Configuration error: Source "in": Unable to open checkpoint file: Permission denied (os error 13)
Sep 04 12:32:49 ip-172-31-16-245 systemd[1]: vector.service: Main process exited, code=exited, status=78/n/a
Sep 04 12:32:49 ip-172-31-16-245 systemd[1]: vector.service: Failed with result 'exit-code'.

[binarylogic]

@Pedneri1 this can also be configured with the data_dir option to ensure Vector writes to a directory it has permissions, but as @bruceg mentioned, it defaults to /var/lib/vector/in.

I'm closing. Feel free to reopen if there is something else going on. Thanks for reporting.

[therus000]

I also got the same error in 0.21.0 version

thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
 2022-04-19T10:12:36.365045Z INFO vector::topology::builder: Healthcheck: Passed.

after while i got this
2022-04-19T10:13:47.296268Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=Read-only file system (os error 30) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving"

any new to fix it?

[spencergilbert]

I also got the same error in 0.21.0 version

thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 thread 'vector-worker' panicked at 'all branches are disabled and there is no else branch', src/kubernetes/reflector.rs:26:9
 2022-04-19T10:12:36.365045Z INFO vector::topology::builder: Healthcheck: Passed.

after while i got this 2022-04-19T10:13:47.296268Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=Read-only file system (os error 30) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving"

any new to fix it?

@therus000 your posted issue seems unrelated to this issue - the panic you shared has been resolved by #12248 and is pending release in 0.21.1

[jszwedko]

Re:

2022-04-19T10:13:47.296268Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=Read-only file system (os error 30) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving"

It does also seem like Vector doesn't have permission to write to the configured data_dir (in addition to the panic bug we are fixing that @spencergilbert mentioned).

[johanseto]

What helped me was changing the user data dir because was owned by root.

This is because the systemctl config for vector service use the vector user.