Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix spark logging and spark tests action workflow #1413

Merged
merged 40 commits into from May 7, 2021
Merged

Fix spark logging and spark tests action workflow #1413

merged 40 commits into from May 7, 2021

Commits on Apr 26, 2021

  1. Test sentry in test setup

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    784c3c7 View commit details
    Browse the repository at this point in the history

  2. Build spark containers before test and use run instead of up

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    dd11f22 View commit details
    Browse the repository at this point in the history

  3. Copy configuration file during spark test run

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    ee3d2b1 View commit details
    Browse the repository at this point in the history

  4. Remove SparkIntegration

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    ba0b805 View commit details
    Browse the repository at this point in the history

  5. Change config.py.sample to hadoop master because it is used in tests

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    c768d5d View commit details
    Browse the repository at this point in the history

  6. Test removing pyspark dependency

    @amCap1712
    amCap1712 committed Apr 26, 2021
    Copy the full SHA
    2b892e5 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2021

  1. Configure python logger

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    46e14a6 View commit details
    Browse the repository at this point in the history

  2. Test a hunch

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    45593bd View commit details
    Browse the repository at this point in the history

  3. Configure base logger

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    03628bd View commit details
    Browse the repository at this point in the history

  4. Add missing logger

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    5e13034 View commit details
    Browse the repository at this point in the history

  5. Move configuration to earlier phase

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    34a272c View commit details
    Browse the repository at this point in the history

  6. Another attempt at logging configuration

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    b006235 View commit details
    Browse the repository at this point in the history

  7. remove pyspark dep

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    910ed62 View commit details
    Browse the repository at this point in the history

  8. Add stop-request-consumer-container.sh script

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    49a91ce View commit details
    Browse the repository at this point in the history

  9. Add metabrainz-spark-test image for use in tests

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    c324ebe View commit details
    Browse the repository at this point in the history

  10. Add back deps

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    0349388 View commit details
    Browse the repository at this point in the history

  11. Copy config file correctly

    @amCap1712
    amCap1712 committed Apr 27, 2021
    Copy the full SHA
    88c25cb View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2021

  1. Fix file path and rearrange

    @amCap1712
    amCap1712 committed Apr 28, 2021
    Copy the full SHA
    76fbc3b View commit details
    Browse the repository at this point in the history

Commits on May 1, 2021

  1. Fix copying config file

    @amCap1712
    amCap1712 committed May 1, 2021
    Copy the full SHA
    dfbd064 View commit details
    Browse the repository at this point in the history

  2. Do not configure sentry in test

    @amCap1712
    amCap1712 committed May 1, 2021
    Copy the full SHA
    2fb6eb9 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2021

  1. Dedup spark Dockerfile

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    75e5f40 View commit details
    Browse the repository at this point in the history

  2. Install development dependencies

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    09ae051 View commit details
    Browse the repository at this point in the history

  3. Remove pyspark dep

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    ca5a5bf View commit details
    Browse the repository at this point in the history

  4. Set PYTHONPATH correctly

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    f476062 View commit details
    Browse the repository at this point in the history

  5. Add py4j to PYTHONPATH

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    7b7c160 View commit details
    Browse the repository at this point in the history

  6. reformat file

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    4c6edbf View commit details
    Browse the repository at this point in the history

  7. Fix SPARK_HOME

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    a27e681 View commit details
    Browse the repository at this point in the history

  8. Second attempt to fix SPARK_HOME

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    77a198b View commit details
    Browse the repository at this point in the history

  9. third attempt to fix SPARK_HOME

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    51de543 View commit details
    Browse the repository at this point in the history

  10. Rearrange schema fields

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    6d7a50a View commit details
    Browse the repository at this point in the history

  11. Rearrange schema fields - 2

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    65d8115 View commit details
    Browse the repository at this point in the history

  12. Rearrange schema fields - 3

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    63edc77 View commit details
    Browse the repository at this point in the history

  13. Rearrange schema fields - 4

    @amCap1712
    amCap1712 committed May 2, 2021
    Copy the full SHA
    245ec5a View commit details
    Browse the repository at this point in the history

Commits on May 5, 2021

  1. Add labels to Dockerfile.spark

    @amCap1712
    amCap1712 committed May 5, 2021
    Copy the full SHA
    f94e95c View commit details
    Browse the repository at this point in the history

  2. Add build-arg to push-request-consumer.sh

    @amCap1712
    amCap1712 committed May 5, 2021
    Copy the full SHA
    f59c161 View commit details
    Browse the repository at this point in the history

  3. Delete obsolete scripts

    @amCap1712
    amCap1712 committed May 5, 2021
    Copy the full SHA
    0276144 View commit details
    Browse the repository at this point in the history

  4. Move remaining spark scripts a level up

    @amCap1712
    amCap1712 committed May 5, 2021
    Copy the full SHA
    036c978 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2021

  1. Add default label to base

    @amCap1712
    amCap1712 committed May 6, 2021
    Copy the full SHA
    5807dd4 View commit details
    Browse the repository at this point in the history

  2. Add build arg after FROM as well

    @amCap1712
    amCap1712 committed May 6, 2021
    Copy the full SHA
    6d0f8e8 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2021

  1. Run spark-request-consumer without docker

    @amCap1712
    amCap1712 committed May 7, 2021
    Copy the full SHA
    ac66b8d View commit details
    Browse the repository at this point in the history