Infra Meeting agenda & minutes

[mscherer]

The infra team has a (internal for now) meeting every 2 mondays.

We decided to list agenda item here.

@deepshikhaaa @rakshithakamath94

[mscherer]

2021-08-09 meeting minutes

present:

• michael
• deepshika
• rakshita

items:

• rakshita: asan job is not working, why ?
  deepshika: disabled for memory leak, long time ago
  misc: also the builder is out, since that's a old one

• rakshita: fix for lcov run this morning, seems old issues coming back
  misc: maybe we should open a ticket
  deepshika: related to python bytecode permissions

• rakshita: fstat survey, only 2 answer, no email
  deepshika: fixing should be low priority
  misc: maybe just change the crontab
  rakshita: will take a look

• deepshika: glusterfs directory empty
  misc: maybe related to latest change on logs compression
  deepshika: need to find the link to illustrate the problem

[mscherer]

2021-08-23 meeting minutes

present:

• deepshika
• michael
• rakshita

items:

• rakshita: linecov debugging. Linecov is now working, but wonder if we can replicate the failure
  propose we run the 2 tests (c8-reg and linecov) back to back on the same builder to see if it was the cause, and investigate
  deepshika: could be some issue with c8-reg cleanup ?
  rakshita: will change jenkins to see if it is reproduce, and we will discuss further

• deepshika: c8-regression. tests are not working, someone looked, then left RH. Wonder about getting faster than t2.medium builders.
  we will discuss further

• deepshika: python-lint
  job started to fail, but that's because it started to work and detect actual errors.
  looking at the job, seems the error is in the script, since it seems to miss a '-prune' to remove the venv
  deepshika will fix the job, then we will see what to do later (eg, fix the error and get to 0, hardcode the maximum number of error, etc)

[mscherer]

2021-09-06 meeting minutes

present:

• rakshita
• misc

items:

• C8 regression failling since friday, by being too slow.
  rakshita investigated, impact is quite high as mgmt is looking at results. Can't reproduce on softserve, but had to disable io_uring.
  misc: various historical slowness cause, kernel, DNS/network, no process to debug that
  c8-2 seemed down, not responding, not rebooting

• linting

actions items

• rakshita will split lint job in 2
• rakshita is looking at fixing shellcheck
• misc install 3rd aws builder for c8
• misc will reinstall c8-2 since it is down
• misc need to document AWS console access, and give it to rakshita
• rakshita will investigate impact of io_uring on test slowness

[mscherer]

2021-09-20 meeting minutes
present:

• raskshitha
• deepshika
• misc

items:

• C8 still slow, investigation didn't yield much results
  rakshitha blocked by softserve timeout of 4h, will get AWS access
  rakshitha also looked around EDNS issue before going in vacation, need to dig further and restart that investigation
• discussions around warning-ng
  deepshika proposed to open a ticket for each linter to signal they are not at 0 warnings

actions items:

• misc will create a account on AWS for rakshitha to not be blocked by softserve
• rakshitha will look at ENDS for C8 builders
• misc to check timing of tests with a script
• rakshitha will ask about keeping or not the linting tests

[mscherer]

2021-10-04 meeting minutes
present:

• misc
• raskshitha

items:

• issue with clang, silently failed (missing deps). Was fixed, but we should have a way to detect that
• coverity check are not working. Issue on the backend, support contacted
• warning-ng followup, requires a jenkins plugins update, so a restart

actions items:

• update of plugins and jenkins planned for friday
• flake8 vs pylint, need to be discussed with devs

[mscherer]

2021-10-18 meeting minutes
present:

• misc
• rakshitha
• deepshikha

items:

• c8-1 failling but not c8-2 or c8-3
  • io_uring need to be disabled ?
  • various people unclear on the status of io_uring on RHEL 8
  • found different kernel version on both kernels, might be related
  • could be fixed with The test case tests/basic/0symbol-check.t is failing the regression test on Centos-8  glusterfs#2865

actions:

• misc: find the command to reboot on older kernel version
• misc: reboot on all builders on the same kernel (latest)
• misc: add uname -a to build-job for easier future debugging
• rakshitha: get more data and open a ticket on the c8 io_uring issue

[mscherer]

No meeting for a while, because PTO, etc. Meeting got moved on the 30 since misc was out for the 29th.

2021-11-30 meeting minutes
present:

• misc
• rakshitha
• deepshikha

items:

• nightly run fail on the multiplex
  • always on the same set tests , but since that's not blocking, seems people do not look
• georep
  • we found that georep tests are being changed (geo-rep test failure with https://github.com/gluster/glusterfs/pull/2945 #148), but context is not clear, and some cleanup are needed in the test (as they modify the config file for ssh)
• still issue on c8-1
  • VM got reinstalled, just in case, should be back tomorrow

actions:

• misc try to get in touch with georep devs to discuss the needs, etc
• rakshitha will contact devs for the multiplex issue
• misc add a meeting for next week

[mscherer]

2021-12-13 meeting minutes

present:

• misc
• rakshitha

items:

• question of 32 bits build

  • seems to only work on F34, due to bug in GCC Glusterfs fail to build with -Werror=stringop-overread on F35 glusterfs#3001
  • rakshitha tested various falgs and will submit PR , also need to remove old flags

• discussion on how to automate jenkins

  • looking at downstream system
  • related to discussion of potential DC move, so there is need to have a way to quickly reinstall jenkins on a AWS VM if needed.

[rakshithakamath94]

Next meeting agenda:

• Discuss about the s390x builder fixes.