table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

[jiangxiaoqiang]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion

---

Question

• Why start oap service start and throw: table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

---

Bug

• Which version of SkyWalking, OS and JRE? Skywalking 7.0.7

• Which company or project?

• What happen?

2020-06-17 08:11:01,022 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -16539 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:04,025 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -19542 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:07,028 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -22545 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:10,031 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -25548 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:13,034 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -28551 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:16,037 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -31554 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:19,040 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -34557 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:22,043 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -37560 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:25,046 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -40563 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:28,049 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -43566 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:31,053 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -46570 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:34,057 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -49574 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:37,060 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -52577 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:40,062 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -55579 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.
2020-06-17 08:11:43,065 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -58582 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

---

Requirement or improvement

• Please describe about your requirements or improvement suggestions.

[wu-sheng]

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

[SxunS]

@wu-sheng Thanks, it did help me.

[john1337]

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

i also met this using skywalking-9.2.0,i started the whole skywalking successfully,but when the k8s cluster reboot,the oap service came up with such "table: alarm_record does not exist. OAP is running in 'no-init' mode" errors, how to fix this?

[liuyujie-tx]

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

i also met this using skywalking-9.2.0,i started the whole skywalking successfully,but when the k8s cluster reboot,the oap service came up with such "table: alarm_record does not exist. OAP is running in 'no-init' mode" errors, how to fix this?

I also have the same problem with skywalking-9.2.0

[leon5233]

Why start oap service start and throw: table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

[leon5233]

Bug
Which version of SkyWalking, OS and JRE? Skywalking 8.9.1 Elasticsearch 7.14

Which company or project?

What happen?

2020-06-17 08:11:01,022 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -43 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

[xuyf1001]

I also have the same problem with skywalking-9.4.0.

[leduy123lsdd]

My steps to fix this issue if you using Helm to install skywalking

1. Get skywalking-oap-init Job template

• execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml

2. Extract Job from manifest.yaml

• in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml

• Delete old job if it existed: kubectl delete -f init-oap.yaml -n skywalking
• Execute the job: kubectl create -f init-oap.yaml -n skywalking

3. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

[kezhenxu94]

My steps to fix this issue if you using Helm to install skywalking

1. Get skywalking-oap-init Job template

• execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml

2. Extract Job from manifest.yaml

• in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml

• Delete old job if it existed: kubectl delete -f init-oap.yaml -n skywalking

• Execute the job: kubectl create -f init-oap.yaml -n skywalking

3. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

No need to do so now. The init job will be executed on installation and upgrade now

[leduy123lsdd]

My steps to fix this issue if you using Helm to install skywalking

1. Get skywalking-oap-init Job template

• execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml

2. Extract Job from manifest.yaml

• in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml

• Delete old job if it existed: kubectl delete -f init-oap.yaml -n skywalking
• Execute the job: kubectl create -f init-oap.yaml -n skywalking

3. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

No need to do so now. The init job will be executed on installation and upgrade now

the problem is when k8s restart it running into 'OAP is running in 'no-init' mode'

[eric-sailfish]

I just tested with the latest helm install, 4.4.0, and this is still an issue.

[leduy123lsdd]

@eric-futureself upgrade skywalking oap and skywalking UI to version 9.5.0 will fix this issue

[anantvardhan04]

Workaround:

You can modify the chart to mimic the run of oap-init Job as an initContainer in UI deployment. This way every time the cluster reboots, when the UI pod comes up, it will create the required tables as part of the init container run, essentially performing the same task as the oap-init Job.
This way OAP pods will come up since all the required tables would have been already created.

Changes:

1. ui-deployment.yaml: Add below two lines under initContainers:

apiVersion: apps/v1
kind: Deployment
.
.
{{- end }}
      initContainers:
   {{- include "skywalking.containers.wait-for-storage" . | nindent 6 }}
   {{- include "skywalking.containers.oap-init-job" . | nindent 6 }}

2. _helpers.tpl: Add template for skywalking.containers.oap-init-job included above

# Template for oap init job container 
{{- define "skywalking.containers.oap-init-job" -}}
- name: oap-init-job
  image: {{ .Values.oap.image.repository }}:{{ required "oap.image.tag is required" .Values.oap.image.tag }}
  imagePullPolicy: {{ .Values.oap.image.pullPolicy }}
  env:
  - name: JAVA_OPTS
    value: "{{ .Values.oap.javaOpts }} -Dmode=init"
    {{- include "skywalking.oap.envs.storage" . | nindent 2 }}
    {{- range $key, $value :=  .Values.oap.env }}
  - name: {{ $key }}
    value: {{ $value | quote }}
    {{- end }}
{{- end -}}

[JangGyeongSik]

I had the same problem, but it was solved thanks to @anantvardhan04
Thanks for the good information :)