Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metricbeat steps to jenkins_build_load_testing.sh #90290

Merged
merged 27 commits into from
Feb 24, 2021
Merged

Add metricbeat steps to jenkins_build_load_testing.sh #90290

Changes from 26 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c39940c
Update jenkins_build_load_testing.sh
marius-dr Feb 4, 2021
48c4a42
Change variables name
marius-dr Feb 4, 2021
13a8bc4
Change to tar.gz package
marius-dr Feb 5, 2021
af94fac
change install location
marius-dr Feb 5, 2021
902beeb
Rename folder
marius-dr Feb 5, 2021
beba1e0
Debug and new folder name
marius-dr Feb 5, 2021
503de07
fix debug
marius-dr Feb 5, 2021
55f9860
Fix path not changed
marius-dr Feb 5, 2021
1a59829
more logging
marius-dr Feb 5, 2021
f6f6a25
add cat logs
marius-dr Feb 5, 2021
37cbaca
more logging and skipping tests for debug
marius-dr Feb 5, 2021
f5238b0
Update jenkins_build_load_testing.sh
marius-dr Feb 5, 2021
b2178b4
Update jenkins_build_load_testing.sh
marius-dr Feb 5, 2021
340d79f
Fix kibana_dir var
marius-dr Feb 5, 2021
1214ee3
Change log file path
marius-dr Feb 5, 2021
0fe0427
Update jenkins_build_load_testing.sh
marius-dr Feb 6, 2021
fe653d9
Update jenkins_build_load_testing.sh
marius-dr Feb 7, 2021
28c06ca
Fix metricbeat start
marius-dr Feb 8, 2021
2f6d58f
Remove extra ls logging
marius-dr Feb 8, 2021
9ff77bb
Changing to MB 7.11.0
marius-dr Feb 15, 2021
3d913ae
Fix package name
marius-dr Feb 15, 2021
71521f5
Fix wrong copy
marius-dr Feb 18, 2021
f960b8b
Add vault credentials for stats
marius-dr Feb 18, 2021
0885e5d
Adding system data and removing log
marius-dr Feb 23, 2021
ee3302b
Merge branch 'master' into marius-dr-add_mb_load_testing
kibanamachine Feb 23, 2021
39e3f5f
Update jenkins_build_load_testing.sh
dmlemeshko Feb 24, 2021
7ab1bb3
Update jenkins_build_load_testing.sh
dmlemeshko Feb 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 55 additions & 3 deletions test/scripts/jenkins_build_load_testing.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,52 @@ if [[ ! "$TASK_QUEUE_PROCESS_ID" ]]; then
./test/scripts/jenkins_xpack_build_plugins.sh
fi

echo " -> Configure Metricbeat monitoring"
# Configure Metricbeat monitoring for Kibana and ElasticSearch, ingest monitoring data into Kibana Stats cluster
# Getting the URL
TOP="$(curl -L http://snapshots.elastic.co/latest/master.json)"
MB_BUILD=$(echo $TOP | sed 's/.*"version" : "\(.*\)", "build_id.*/\1/')
echo $MB_BUILD
MB_BUILD_ID=$(echo $TOP | sed 's/.*"build_id" : "\(.*\)", "manifest_url.*/\1/')

URL=https://snapshots.elastic.co/${MB_BUILD_ID}/downloads/beats/metricbeat/metricbeat-${MB_BUILD}-linux-x86_64.tar.gz
URL=https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.11.0-linux-x86_64.tar.gz
echo $URL
Comment on lines +26 to +28
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marius-dr , is it something we should take care in future? Do we need to upgrade later?
I wonder if we need to open an issue for it in elastic/kibana-load-testing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Until we release 8.0.0, this should be compatible 100%. So that's 1 year from now at least.

# Downloading the Metricbeat package
while [ 1 ]; do
wget -q --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -t 0 --continue --no-check-certificate --tries=3 $URL
if [ $? = 0 ]; then break; fi; # check return value, break if successful (0)
sleep 1s;
done;

# Install Metricbeat
echo "untar metricbeat and config"
#tar -xzf metricbeat-${MB_BUILD}-linux-x86_64.tar.gz
tar -xzf metricbeat-7.11.0-linux-x86_64.tar.gz
#--directory "$KIBANA_DIR"
ls -l
echo "rename"
#mv metricbeat-${MB_BUILD}-linux-x86_64 metricbeat-install
mv metricbeat-7.11.0-linux-x86_64 metricbeat-install
ls -l
# Configure Metricbeat

echo " -> Changing metricbeat config"
pushd ../kibana-load-testing
cp cfg/metricbeat/elasticsearch-xpack.yml $KIBANA_DIR/metricbeat-install/modules.d/elasticsearch-xpack.yml
cp cfg/metricbeat/kibana-xpack.yml $KIBANA_DIR/metricbeat-install/modules.d/kibana-xpack.yml
echo "fields.build: ${BUILD_ID}" >> cfg/metricbeat/metricbeat.yml
echo "path.config: ${KIBANA_DIR}/metricbeat-install" >> cfg/metricbeat/metricbeat.yml
echo "cloud.auth: ${USER_FROM_VAULT}:${PASS_FROM_VAULT}" >> cfg/metricbeat/metricbeat.yml
cp cfg/metricbeat/metricbeat.yml $KIBANA_DIR/metricbeat-install/metricbeat.yml
# Disable system monitoring: enabled for now to have more data
#mv $KIBANA_DIR/metricbeat-install/modules.d/system.yml $KIBANA_DIR/metricbeat-install/modules.d/system.yml.disabled
popd

# doesn't persist, also set in kibanaPipeline.groovy
export KBN_NP_PLUGINS_BUILT=true

echo " -> building and extracting default Kibana distributable for use in functional tests"
echo " -> Building and extracting default Kibana distributable for use in functional tests"
cd "$KIBANA_DIR"
node scripts/build --debug --no-oss
linuxBuild="$(find "$KIBANA_DIR/target" -name 'kibana-*-linux-x86_64.tar.gz')"
Expand All @@ -29,11 +71,21 @@ tar -xzf "$linuxBuild" -C "$installDir" --strip=1
mkdir -p "$WORKSPACE/kibana-build-xpack"
cp -pR install/kibana/. $WORKSPACE/kibana-build-xpack/

echo " -> test setup"
echo " -> Setup env for tests"
source test/scripts/jenkins_test_setup_xpack.sh

echo " -> run gatling load testing"
# Start Metricbeat
echo " -> Starting metricbeat"
pushd $KIBANA_DIR/metricbeat-install
nohup ./metricbeat > metricbeat.log 2>&1 &
popd

echo " -> Running gatling load testing"
export GATLING_SIMULATIONS="$simulations"
node scripts/functional_tests \
--kibana-install-dir "$KIBANA_INSTALL_DIR" \
--config test/load/config.ts

# Show output of Metricbeat. Disabled. Enable for debug purposes
#echo "output of metricbeat.log"
#cat $KIBANA_DIR/metricbeat-install/metricbeat.log