HPCC-24970 Sasha K8s services. #14536

jakesmith · 2021-02-01T18:26:12Z

Signed-off-by: Jake Smith jake.smith@lexisnexisrisk.com

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

hpcc-jirabot · 2021-02-01T18:26:22Z

https://track.hpccsystems.com/browse/HPCC-24970
Jira updated

AttilaVamos · 2021-02-01T19:02:09Z

#MsgID    Audience Date       Time(milli)  PID   TID   Quoted
00000000 PRG 2021-02-01 18:44:10.962 21111 21111 "Build community_7.13.0-trunk0[heads/master-0-g243c26]"
00000001 PRG 2021-02-01 18:44:10.968 21111 21111 "Backtrace:"
00000002 PRG 2021-02-01 18:44:10.968 21111 21111 "  /opt/HPCCSystems/lib/libjlib.so(_Z16printStackReportx+0x45) [0x7eff47f9b485]"
00000003 PRG 2021-02-01 18:44:10.968 21111 21111 "  /opt/HPCCSystems/lib/libjlib.so(_Z20raiseAssertExceptionPKcS0_j+0x83) [0x7eff47f9b713]"
00000004 PRG 2021-02-01 18:44:10.968 21111 21111 "  saserver() [0x418948]"
00000005 PRG 2021-02-01 18:44:10.968 21111 21111 "  saserver() [0x409d51]"
00000006 PRG 2021-02-01 18:44:10.968 21111 21111 "  /lib64/libc.so.6(__libc_start_main+0xf5) [0x7eff4441d505]"
00000007 PRG 2021-02-01 18:44:10.968 21111 21111 "  saserver() [0x40a391]"
00000008 PRG 2021-02-01 18:44:10.968 21111 21111 "ERROR: assert(!sashaSDSCoalescingServer) failed - file: sacoalescer.cpp, line 288"
00000009 USR 2021-02-01 18:44:10.968 21111 21111 "ERROR: 3000: saserver.cpp(506) : Sasha Server Exception:  : assert(!sashaSDSCoalescingServer) failed - file: sacoalescer.cpp, line 288"
0000000A USR 2021-02-01 18:44:10.969 21111 21111 "Stopping 0"

jakesmith · 2021-02-02T10:59:57Z

@wangkx - please review, in particular, the esp service changes to pick up the sasha service. This will relate to: https://track.hpccsystems.com/browse/HPCC-24968

jakesmith · 2021-02-02T11:40:49Z

@ghalliday - please review

richardkchapman

Couple of minor issues

testing/helm/run.sh

jakesmith · 2021-02-02T11:51:44Z

@richardkchapman - removed spurious changes to testing/helm/run.sh

dali/sasha/saarch.cpp

ghalliday

@jakesmith looks good. A few minor comments.

dali/sasha/saarch.cpp

dali/sasha/saserver.cpp

dali/sasha/saxref.cpp

esp/smc/SMCLib/TpWrapper.cpp

helm/hpcc/templates/_helpers.tpl

ghalliday · 2021-02-03T11:16:30Z

helm/hpcc/templates/dali.yaml

+{{- include "hpcc.addSashaVolumes" (dict "root" $ "me" .) | indent 6 }}
+{{- end }}
+{{- end }}
+{{ include "hpcc.addDaliVolume" $commonCtx | indent 6 }}
 {{ include "hpcc.addSecretVolumes" (dict "root" $ "categories" (list "system" ) ) | indent 6 }}


This will likely need to include storage secrets (e.g. for xref, and possibly other uses)

It will depend on the services included in Dali's pod.
The expectation is that it will just be the coalescer in the Dali pod, but if it's e.g. the archiver, or file-expiry, or xref, then it would need "storage" too. So I should detect and make conditional.

Will fix.

jakesmith · 2021-02-04T10:52:08Z

@ghalliday - please see new commit

ghalliday

@jakesmith a few minor comments. I'm not confident I have reviewed the helm files fully.

esp/smc/SMCLib/TpWrapper.cpp

helm/hpcc/templates/_helpers.tpl

ghalliday · 2021-02-04T12:18:33Z

helm/hpcc/templates/dali.yaml

 {{ include "hpcc.addSecretVolumeMounts" (dict "root" $ "categories" (list "system" ) ) | indent 8 }}
+{{- range .services -}}


There should be an entry in the config for dali (and all sasha config files) that contains the vault information. Mentioned in Shamser's PR as well.

there are no secrets/vaults generated for Dali in master at the moment.
I don't think Dali has needed any, until now with introduction of sasha services that may.

I can wait though for Shamser's PR to be merged, so it doesn't clash etc.., then use a common list.

ghalliday

I think good from my perspective, but I would like someone else's (@richardkchapman ?) comments on the Helm changes.

ghalliday · 2021-02-05T11:30:56Z

helm/hpcc/templates/dali.yaml

 {{ include "hpcc.addSecretVolumes" (dict "root" $ "categories" (list "system" ) ) | indent 6 }}
+{{- if or (has "data" .access) (has "dalidata" .access) -}}
+{{ include "hpcc.addSecretVolumes" (dict "root" .root "categories" (list "storage" ) ) | indent 6 -}}


Should really be using a single call with a conditional list. If everything else is ok, then it can be merged and @shamser fix it in his PR.

ghalliday

@jakesmith one comment. I think ready for squashing and a final review.

dali/sasha/saxref.cpp

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>

jakesmith · 2021-02-10T11:34:19Z

@ghalliday - squashed and rebased to latest master.

HPCCSmoketest · 2021-02-10T12:10:42Z

Automated Smoketest: ✅
OS: centos 7.6.1810 (Linux 3.10.0-957.1.3.el7.x86_64)
GCC: gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Host: ip-10-20-0-6.ca-central-1.compute.internal
Sha: c0faec3
Build: success
Milestone:Install hpccsystems-platform-community_7.13.0-trunk0.el7.x86_64.rpm
HPCC Start: OK

Unit tests result:

Test	total	passed	elaps
unittest	146	146	54 sec
wutoolTest(Dali)	19	19	0 sec
wutoolTest(Cassandra)	19	19	6 sec

Regression test result:

phase	total	pass	elaps
setup (hthor)	9	9	22 sec (00:00:22)
setup (thor)	9	9	41 sec (00:00:41)
setup (roxie)	19	19	22 sec (00:00:22)
test (hthor)	954	954	684 sec (00:11:24)
test (thor)	865	865	1163 sec (00:19:23)
test (roxie)	1027	1027	802 sec (00:13:22)

HPCC Stop: OK
HPCC Uninstall: OK
Time stats:

Prep time	Build time	Package time	Install time	Start time	Test time	Stop time	Summary
9 sec (00:00:09)	620 sec (00:10:20)	128 sec (00:02:08)	20 sec (00:00:20)	17 sec (00:00:17)	1475 sec (00:24:35)	17 sec (00:00:17)	2286 sec (00:38:06)

jakesmith force-pushed the hpcc-24970-sasha-services branch from 78aec5e to 243c26e Compare February 1, 2021 18:30

jakesmith force-pushed the hpcc-24970-sasha-services branch 3 times, most recently from 6fe1503 to 59791f2 Compare February 2, 2021 09:29

jakesmith requested review from wangkx and richardkchapman February 2, 2021 10:58

jakesmith requested a review from ghalliday February 2, 2021 11:40

richardkchapman requested changes Feb 2, 2021

View reviewed changes

testing/helm/run.sh Outdated Show resolved Hide resolved

testing/helm/run.sh Outdated Show resolved Hide resolved

jakesmith force-pushed the hpcc-24970-sasha-services branch from 59791f2 to c2a855e Compare February 2, 2021 11:51

jakesmith force-pushed the hpcc-24970-sasha-services branch from c2a855e to 8144da1 Compare February 2, 2021 12:56

jakesmith requested a review from richardkchapman February 3, 2021 09:25

ghalliday reviewed Feb 3, 2021

View reviewed changes

dali/sasha/saarch.cpp Outdated Show resolved Hide resolved

ghalliday reviewed Feb 3, 2021

View reviewed changes

jakesmith requested a review from ghalliday February 4, 2021 10:51

ghalliday reviewed Feb 4, 2021

View reviewed changes

jakesmith force-pushed the hpcc-24970-sasha-services branch from 6005f93 to b7094b9 Compare February 4, 2021 13:16

ghalliday previously approved these changes Feb 5, 2021

View reviewed changes

jakesmith dismissed ghalliday’s stale review via c71be55 February 5, 2021 14:15

ghalliday reviewed Feb 10, 2021

View reviewed changes

dali/sasha/saxref.cpp Outdated Show resolved Hide resolved

HPCC-24970 Sasha K8s services.

c0faec3

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>

jakesmith force-pushed the hpcc-24970-sasha-services branch from c71be55 to c0faec3 Compare February 10, 2021 11:27

jakesmith requested a review from ghalliday February 10, 2021 11:34

richardkchapman approved these changes Feb 10, 2021

View reviewed changes

ghalliday approved these changes Feb 10, 2021

View reviewed changes

ghalliday merged commit fc08bc8 into hpcc-systems:master Feb 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-24970 Sasha K8s services. #14536

HPCC-24970 Sasha K8s services. #14536

jakesmith commented Feb 1, 2021 •

edited

hpcc-jirabot commented Feb 1, 2021

AttilaVamos commented Feb 1, 2021

jakesmith commented Feb 2, 2021

jakesmith commented Feb 2, 2021

richardkchapman left a comment

jakesmith commented Feb 2, 2021

ghalliday left a comment

ghalliday Feb 3, 2021

jakesmith Feb 3, 2021

jakesmith commented Feb 4, 2021

ghalliday left a comment

ghalliday Feb 4, 2021

jakesmith Feb 4, 2021

ghalliday left a comment

ghalliday Feb 5, 2021

ghalliday left a comment

jakesmith commented Feb 10, 2021

HPCCSmoketest commented Feb 10, 2021

		{{ include "hpcc.addSecretVolumeMounts" (dict "root" $ "categories" (list "system" ) ) \| indent 8 }}
		{{- range .services -}}

Navigation Menu

HPCC-24970 Sasha K8s services. #14536

HPCC-24970 Sasha K8s services. #14536

Conversation

jakesmith commented Feb 1, 2021 • edited

Type of change:

Checklist:

Smoketest:

Testing:

hpcc-jirabot commented Feb 1, 2021

AttilaVamos commented Feb 1, 2021

jakesmith commented Feb 2, 2021

jakesmith commented Feb 2, 2021

richardkchapman left a comment

Choose a reason for hiding this comment

jakesmith commented Feb 2, 2021

ghalliday left a comment

Choose a reason for hiding this comment

ghalliday Feb 3, 2021

Choose a reason for hiding this comment

jakesmith Feb 3, 2021

Choose a reason for hiding this comment

jakesmith commented Feb 4, 2021

ghalliday left a comment

Choose a reason for hiding this comment

ghalliday Feb 4, 2021

Choose a reason for hiding this comment

jakesmith Feb 4, 2021

Choose a reason for hiding this comment

ghalliday left a comment

Choose a reason for hiding this comment

ghalliday Feb 5, 2021

Choose a reason for hiding this comment

ghalliday left a comment

Choose a reason for hiding this comment

jakesmith commented Feb 10, 2021

HPCCSmoketest commented Feb 10, 2021

jakesmith commented Feb 1, 2021 •

edited