Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MWT-26 #576

Draft
wants to merge 60 commits into
base: master
Choose a base branch
from
Draft

MWT-26 #576

wants to merge 60 commits into from

Commits on Apr 15, 2020

  1. Add visual separations for the script's sections.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    ef3c10d View commit details
    Browse the repository at this point in the history

  2. Make container name be dependent on test parameter file name.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    70579ca View commit details
    Browse the repository at this point in the history

  3. Check package repos every time, create Marathon group.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    9564118 View commit details
    Browse the repository at this point in the history

  4. Use container name for docker image tag and run directory.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    287a9b9 View commit details
    Browse the repository at this point in the history

  5. Automatically format some files with Black.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    04717bd View commit details
    Browse the repository at this point in the history

  6. Parameterize group file name, run commands in container.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    9fbca5b View commit details
    Browse the repository at this point in the history

  7. Run commands in the container.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    0e856a2 View commit details
    Browse the repository at this point in the history

  8. Quote this.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    d9a7e96 View commit details
    Browse the repository at this point in the history

  9. Make deploy-dispatchers.py script support group roles.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    585beac View commit details
    Browse the repository at this point in the history

  10. Show CLUSTER_URL in pre-run report.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    c4d870f View commit details
    Browse the repository at this point in the history

  11. Add group_role support to streaming workload deploy script.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    cd95b27 View commit details
    Browse the repository at this point in the history

  12. Make sure service name is prefixed with a slash.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    5c1c912 View commit details
    Browse the repository at this point in the history

  13. Add group_role support to the batch_test.py script.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    f40351a View commit details
    Browse the repository at this point in the history

  14. Add required roles and permissions required for group role enforcement.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    0b25ba5 View commit details
    Browse the repository at this point in the history

  15. Add spark-options.json file.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    0a2f2d1 View commit details
    Browse the repository at this point in the history

  16. Typo. 

    Co-Authored-By: Sam Tran <stran@mesosphere.com>
    @mpereira @samvantran @rishabh96b
    2 people authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    2bd9407 View commit details
    Browse the repository at this point in the history

  17. Add CLI parameter description. 

    Co-Authored-By: Sam Tran <stran@mesosphere.com>
    @mpereira @samvantran @rishabh96b
    2 people authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    73b5824 View commit details
    Browse the repository at this point in the history

  18. Remove failing jobs stuff.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    bf0fcf0 View commit details
    Browse the repository at this point in the history

  19. Fix shellcheck warning.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    0025ef4 View commit details
    Browse the repository at this point in the history

  20. Variable renames, shellcheck fixes, DSEngine, total cpu/mem/gpu.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    50b56ef View commit details
    Browse the repository at this point in the history

  21. GROUP_NAME should be coming with no slash prefix.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    b849f5d View commit details
    Browse the repository at this point in the history

  22. Create quota if it doesn't exit.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    2f1075b View commit details
    Browse the repository at this point in the history

  23. Install recent DC/OS CLI.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    222cbe2 View commit details
    Browse the repository at this point in the history

  24. Fix quoting.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    464a17b View commit details
    Browse the repository at this point in the history

  25. Should be group name here.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    af9129e View commit details
    Browse the repository at this point in the history

  26. Make revoke_permissions() also take role list.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    c2daf5e View commit details
    Browse the repository at this point in the history

  27. Fix indentation.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    65b3fd9 View commit details
    Browse the repository at this point in the history

  28. Make DC/OS CLI binary a parameter.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    bec6c9f View commit details
    Browse the repository at this point in the history

  29. Don't break out of loop, just skip to the next element.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    5b34f05 View commit details
    Browse the repository at this point in the history

  30. This was breaking.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    3cd62ec View commit details
    Browse the repository at this point in the history

  31. Add script to list service tasks.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    b9fa2ec View commit details
    Browse the repository at this point in the history

  32. service_options was being used before being set.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    1931abb View commit details
    Browse the repository at this point in the history

  33. Fix environment variable name.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    956200c View commit details
    Browse the repository at this point in the history

  34. installing jupiter

    @alexeygorobets @rishabh96b
    alexeygorobets authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    976a775 View commit details
    Browse the repository at this point in the history

  35. Improve DSEngine workload deployment script and options.

    @mpereira @rishabh96b
    mpereira authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    5d25c89 View commit details
    Browse the repository at this point in the history

  36. rename dsengine options to dsengine-options.jso

    @alexeygorobets @rishabh96b
    alexeygorobets authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    78904b7 View commit details
    Browse the repository at this point in the history

  37. rename dsengine options to dsengine-options.json

    @alexeygorobets @rishabh96b
    alexeygorobets authored and rishabh96b committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    275da0f View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2020

  1. add MWT-25 dry run and MWT-25 configs

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    c52f1ed View commit details
    Browse the repository at this point in the history

  2. update cluster url for MWT26

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    ae341c2 View commit details
    Browse the repository at this point in the history

  3. update kafka version to 2.10.0-5.5.1-beta

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    b474d78 View commit details
    Browse the repository at this point in the history

  4. enable exernal volumes on kafka brokers

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    6cd6997 View commit details
    Browse the repository at this point in the history

  5. update confluent-zk version to 2.8.0-5.5.1-beta

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    030a3cf View commit details
    Browse the repository at this point in the history

  6. update Cassandra to 2.10.0-3.11.6-beta

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    ba74e60 View commit details
    Browse the repository at this point in the history

  7. add external volumes to cassandra options

    @alexeygorobets
    alexeygorobets committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    05743fa View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2020

  1. use latest spark 2.11.0-2.4.6

    @rishabh96b
    rishabh96b committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    610e6a3 View commit details
    Browse the repository at this point in the history

  2. update spark stub to universe-converter

    @rishabh96b
    rishabh96b committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    6888ecc View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dcos-58437-deploy-workloads-under-role-enforced-group' … 

    …into mwt-26

# Conflicts:
#	spark-testing/spark_utils.py
#	testing/sdk_install.py
    @alexeygorobets
    alexeygorobets committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    3a4b36e View commit details
    Browse the repository at this point in the history

  4. Adds data science engine configs

    @farhan5900
    farhan5900 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    6ffb082 View commit details
    Browse the repository at this point in the history

  5. update configs for dry run

    @alexeygorobets
    alexeygorobets committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    288b97a View commit details
    Browse the repository at this point in the history

  6. use root user for services

    @alexeygorobets
    alexeygorobets committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    853dbe5 View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'origin/mwt-26' into mwt-26 

    # Conflicts:
#	scale-tests/configs/dsengine-options.json
#	scale-tests/run.sh
    @alexeygorobets
    alexeygorobets committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    f087a33 View commit details
    Browse the repository at this point in the history

  8. Grant task:user:root permissions on master and agent for DSE.

    @kaiwalyajoshi
    kaiwalyajoshi committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    c8e8fdc View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2020

  1. install DSE with other services

    @alexeygorobets
    alexeygorobets committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    7d85cbb View commit details
    Browse the repository at this point in the history

  2. update dsengine options

    @alexeygorobets
    alexeygorobets committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    8b71331 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'mwt-26' of https://github.com/mesosphere/spark-build in… 

    …to mwt-26
    @alexeygorobets
    alexeygorobets committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    a8c6e56 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. fix DSE permissions

    @alexeygorobets
    alexeygorobets committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    083ebb9 View commit details
    Browse the repository at this point in the history

  2. add MWT 26 config

    @alexeygorobets
    alexeygorobets committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    2750a25 View commit details
    Browse the repository at this point in the history

  3. not using GPU in DSE

    @alexeygorobets
    alexeygorobets committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    6032efa View commit details
    Browse the repository at this point in the history

  4. fix marathon check for batched job

    @alexeygorobets
    alexeygorobets committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    aaace06 View commit details
    Browse the repository at this point in the history

  5. use right branch in batched workload for spark

    @alexeygorobets
    alexeygorobets committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    d34af99 View commit details
    Browse the repository at this point in the history