Skip to content

Conversation

sidsatyelp
Copy link
Member

@sidsatyelp sidsatyelp commented Jun 4, 2025

What this change does

  1. Extracts the jira_ticket handling to a separate private method. We still run all tests through the public interface get_spark_conf()
  2. Adds a yelp-clog requirement to tox that is conditionally installed.
  3. When flag spark.yelp.jira_ticket.enabled is false in srv-configs, it logs a warning to monk with the user param passed to get_spark_conf(). Ensures that paasta validate and paasta m-f-d work as usual.
  4. When the flag is true, it will block non-exempt users
  5. Add jenkins to allowed users list (Proof that this user is indeed called jenkins)

Testing

When flag is disabled

# should launch with no warning but write to monk
paasta spark-run --aws-profile=dev-cloud-economics --cmd "spark-submit /code/integration_tests/s3.py"
# output: https://fluffy.yelpcorp.com/i/xZstCSFXx0L4sKBQnThv2p42DnC7vMZn.html

scribereader -e devc spark_jira_ticket -n 100
{"timestamp": 1749771624, "event": "jira_ticket_validation_warning", "level": "WARNING", "reason": "Ticket missing or invalid", "user": "sids", "jira_ticket_provided": "BLAH-sids", "paasta_cluster": "pnw-devc-spark", "paasta_service": "spark", "paasta_instance": "adhoc_sids_spark-submit"}

# should launch with labels attached to pods
paasta spark-run --aws-profile=dev-cloud-economics --jira-ticket=CLOUD-802 --cmd "spark-submit /code/integration_tests/s3.py"
# output: https://fluffy.yelpcorp.com/i/kLSxKxmtk2qB10mh2TTm123ZSHkCh8KN.html

# paasta validate works
╰─ paasta validate -y ~/repos/yelpsoa-configs -s spark
✓ Successfully validated schema: tron-pnw-prod.yaml
✓ Successfully validated schema: tron-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-prod.yaml
✓ Successfully validated schema: eks-pnw-prod.yaml
✓ Successfully validated schema: eks-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-prod-spark.yaml
✓ Successfully validated schema: adhoc-norcal-devc.yaml
✓ Successfully validated schema: service.yaml
✓ Successfully validated schema: kubernetes-pnw-prod.yaml
✓ Successfully validated schema: kubernetes-pnw-devc.yaml
✓ tron-pnw-prod.yaml is valid.
✓ tron-pnw-devc.yaml is valid.
✓ All PaaSTA Instances for are valid for all clusters
✓ All spark's instance names in cluster norcal-devc are unique
✓ All spark's instance names in cluster norcal-stagef are unique
✓ All spark's instance names in cluster pnw-devc are unique
✓ All spark's instance names in cluster pnw-prod are unique
✓ All spark's instance names in cluster pnw-prod-spark are unique
✓ No orphan secrets found

When flag is enabled

# should not launch
paasta spark-run --aws-profile=dev-cloud-economics --cmd "spark-submit /code/integration_tests/s3.py"
# output: https://fluffy.yelpcorp.com/i/qp6Q4200DsVGl3t9FKp7n2bMGt8l5mZ9.html

# should launch with labels attached to pods
paasta spark-run --aws-profile=dev-cloud-economics --jira-ticket=CLOUD-802 --cmd "spark-submit /code/integration_tests/s3.py"
# output:  https://fluffy.yelpcorp.com/i/N40kkpV06BqpXGvvPWNsFlcdwrFk5Pn3.html

# paasta validate works 
paasta validate -y ~/repos/yelpsoa-configs -s spark
✓ Successfully validated schema: tron-pnw-prod.yaml
✓ Successfully validated schema: tron-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-prod.yaml
✓ Successfully validated schema: eks-pnw-prod.yaml
✓ Successfully validated schema: eks-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-devc.yaml
✓ Successfully validated schema: adhoc-pnw-prod-spark.yaml
✓ Successfully validated schema: adhoc-norcal-devc.yaml
✓ Successfully validated schema: service.yaml
✓ Successfully validated schema: kubernetes-pnw-prod.yaml
✓ Successfully validated schema: kubernetes-pnw-devc.yaml
✓ tron-pnw-prod.yaml is valid.
✓ tron-pnw-devc.yaml is valid.
✓ All PaaSTA Instances for are valid for all clusters
✓ All spark's instance names in cluster norcal-devc are unique
✓ All spark's instance names in cluster norcal-stagef are unique
✓ All spark's instance names in cluster pnw-devc are unique
✓ All spark's instance names in cluster pnw-prod are unique
✓ All spark's instance names in cluster pnw-prod-spark are unique
✓ No orphan secrets found

@sidsatyelp sidsatyelp requested a review from nemacysts June 12, 2025 02:02
@chi-yelp
Copy link
Contributor

Looks good to me on the implementation side, will let @nemacysts to confirm the repo setup

Copy link
Member

@nemacysts nemacysts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(i'm just requesting changes since i want to make sure we don't merge this with the requirements/dependencies as-is)

@sidsatyelp sidsatyelp requested review from chi-yelp and nemacysts June 12, 2025 23:43
@sidsatyelp
Copy link
Member Author

@nemacysts Can you review these changes please?

@sidsatyelp sidsatyelp requested a review from chi-yelp June 13, 2025 16:41
tox.ini Outdated
-rrequirements-dev.txt
-rrequirements.txt
commands=
- pip install yelp-clog
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would probably be nice to have a comment here about why this is being installed - but since this is temporary i won't be too picky :p

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the comment.

@sidsatyelp sidsatyelp requested a review from nemacysts June 13, 2025 16:56
@sidsatyelp sidsatyelp merged commit e795919 into master Jun 13, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants