disable Spark INFO output by default #41
Comments
|
Partially done: https://github.com/cseed/k3/commit/3cfd6b226ec1bd885faa1fae96965435dd78130b There is still a small number of messages I haven't figured out how to disable. |
|
This was resolved some time ago. |
cseed
added a commit
to cseed/hail
that referenced
this issue
Sep 22, 2018
danking
added a commit
to danking/hail
that referenced
this issue
Sep 24, 2018
danking
added a commit
that referenced
this issue
Sep 24, 2018
* initial commit * approved check seems to work * add environment * readme with helpful tips * more developer info * better error message when oauth-token is missing * more dev hints * abstract over repo * add git ignore * make amenable to packaging * add design doc * bunch of changes * fixes * note about secret * add test * document envs * update design doc * a bunch of fixes * notes about service account iam * update hashes to facilitate a test * pull batch server url into a const * fake pr build script * setup for secrets * fix pr-build-script * document new batch volumes * some halfway working setup * wip * refactor to state manipulate + heal model * a variety of bugs and a couple todos * i think this actually sets the metadata? * bunch of fixes, introduce polling thread * locks and fix github state check * no threading, restore popped status * done? * real build script * fix message about all prs being tested or running * first attempt at dockerfile * env variables, move secrets, fix invalidated prs logic * get docker file working mostly * fix pr-build-script * two little fixes * update dockerignore * expose port 5000 * expose to the world * be resilient to bad or out of date job ids * reduce cpu request * first pass on deployment * add info about external ips * fix pycache ignore * fix test report url * refresh from batch, catch cancel fails, slow down refresh interval * logging * fix dockerignore * give user ownership of current directory as well * a little clean up and better logic for refreshing batch state * fix bad variable reference in job cancellation during github state refresh * do not blow away review state on push * note which target we are talking about when all prs are tested or running * be resilient to jobs not existing * more logging in test_pr * convert to a format string * start publishing bundles * Revert "start publishing bundles" This reverts commit 63b2036. * watched repos is customizable * fixes * shit in an attempt to use docker but ugh * note firewalls rules thing * fix url to repo RE also ignore unwatched repos * a few fixes * ignore pycache * add makefile * ignore more stuff * new tests * fix bad variable reference * note buildable repos * work to generalize from just the hail repo * fix buildable repos * better debugging into when test fails * fix tests * fix build script, TARGET_BRANCH might be ambiguous * be robust to missing artifacts * properly log exception * use copy, drop job_id when cancelled job * quietly throw away orphan heads * tag images with git sha * cancel jobs on push, fix copy, and give merge a message * fix build script oop * add test with master change * add test for approved prs * add dependency on pytest * clear job id when setting state to pending * fix deployment.yaml env var * need to re-fetch origin as well * better assertion failure messages for dict subset * add description of github-tokens directory * ignore temporary git directory created by tests * ignore the testing oauth tokens * ignore gcloud token as well * sort .gitignore * also ignore .cache and .pytest_cache * many improvements and fixes to tests * god, SHUT up! * fix new token system * fixes and add failing test for mergedness * clean up get/post logic * lots of fixes * self. on every field * add more sophisticated matching * merged_and_old wrong identation * fix a couple bugs in new match stuff * fix match again * add json_response to *_repo * add json_response * ad dlog, fix gc syntax errors * work towards fixing push test since branch is protected now * more info on testing * better error messages and fix bug * bunch of fixes * add fixme * fixes * fix * mor elogging around github state refresh * a couple fixes to the tests * more logging, less logging, use VERSION_FOR_TYPE VERSION_FOR_TYPE is used to version the pod type attribute so other CI instances dont stomp on my pods * add gc field to Status.to_json * clean up PRs for targets with no PRs in GH * better logging, fix dead targets logic * fix confusing double definition * fix syntax * rely on finally so we dont have so many errors * fix make file for docker tag * add auto merge flag and dont blow away merged statuses after push * remove nesting level in artifacts copy * fix misspelled variable name * try to hide git output from log * actually sleep inbetween polls * warn is deprecated * remove VERSION_FOR_TYPE * update deployment for latest CI image * stop spamming the log with job logs * update deployment * do not hide git stdout and stderr * configure username and email before merge * update deployment * be resilient to merge conflicts in the docker_image * update deployment * bugssss * update deployment * use merge instead of rebase to avoid issues * update dployment * fix deployment I accidentally used an old commit hash * also update docker_image on force_retest * update deployment * more status logging * update deployment * fix missing parens * update deployment * more logging and fix reading status in GH refresh * update deployment * also check for file not found error * update deployment * do not allow max statuses to break batch_state_refresh * update deployment * avoid failing when 422 is received * update deployment * kind of handle the too many statuses situation? * update CONTEXT to avoid status limits * update deployment * add timeouts to all requests * add assertion about attributes and target_url * update deployment * implement pagination * update deployment * do not log link headers, longer timeouts on self check * update deployment.yaml * be robust to changes in job attribute schema * update deployment * add hail-is/batch to watched repos (#25) * add healthcheck * add livenessprobe * fix bug in forgetting of old PRs * update deployment * fix port * build should fail if artifacts exist but are not copied (#32) * Cleanup (#35) * abstraciton! * wip wip remove old ci remove old tests rename newci to ci remove unused funciton add heal fixes rename http wip take oldest PR, I think i think no more caching there are two different github urls fixes fixxxxxes non global creds setting fixes fixes two tiny fixes bunch of fixes run formatter run formatter formatting fix PR.from_json formatting, PEP8 formatting and fixes fixes lots of fixes never return none relax and strengthen assertions fixes thread json_response throug http_helper get oauth tokens right fix user fix token fixes passes tests naming deploying fixes deploy script more deploy things add healthcheck add liveness probe to deployment sleep before checking for artifact deployability is configurable also fix pr build script in new branch rename constants and real_constants variety of cleanups to get test-locally.sh working again bugs fix bug: GitHubPR has no .target * try to spam the log less * aggressively use short_str in log statements * trailing semi-colons visually distinguish the end * unify logging a bit * make jobs logging nicer * fix parse error * do not duplicate messages * forgot to include json * few more logging fixups * a little more context for PR and GHPR * fix bug: eagerly update build status If the PR is found during review update before a pull_request push request is received, we will wait until the next heal to build. Instead, we should eagerly build now (the later pull_request webhook will not build because the shas are up to date). ] * dont deploy non deployable refs also default to SHA_LENGTH * few more logging cleanups * move tests to another folder * update deployment * also copy in pr-deploy-script (#36) * update deployment.yaml (#37) * fix ham finger (#38) * update deployment (#39) * Fix NoImage constructor (#40) * update deployment * fix NoImage constructor * update deployment for real (#41) * more logging around dead prs (#42) * more logging around dead prs * update deployment.yaml * filter by target repo (#44) * update deployment (#45) * useful shell functions (#46) * Fix JSON Parsing & Target SHA Refresh (#49) * fix json parsing in force endpoints * fix target sha updating too * update deployment * relax build transition assertion (#50) * relax build transition assertion * update deployment * update deployment * fix target sha update to hail-ci-build-image (#51) * Fix build state again (#53) * ands and ors * update deployment * enable deployment of master (#55) * Deploy Index Missing (#57) * fix dockerfile to include deploy-index * update deployment * fix type error (#58) * d is already json * update deployment * Redeploy takes a SHA not a Ref (#60) * redeploy takes a sha * update deployment * Fix PR Dismissal (#61) * fix pr dismissal * update deployment * fix type error (#62) * fix type error * update deployment * disable deployment on master (#63) * deploy better (#64) * deploy better * also print latest_deployed in status * update deployment * fix bug (#66) * deploy master * sometimes attributes is missing * update deployment * Update deployment.yaml * update deployment * Configurable deploying (#67) * add set_deployable and fix bug in deploy_build_finished * fix * update deployment * Eagerly merge reviewed prs (#52) * eagerly heal if review change makes pr mergeable * update deployment * heal each repo once (#68) * repos might be duplicated, so use a set * update deployment * only log when there is something to forget * deploy master (#65) * delete jobs after refreshing so we can run new ones (#69) * delete jobs after refreshing so we can run new ones * update deployment * fix refresh (#70) * only delete if completed or cancelled * update deployment * mount docker sock in deploy job (#71) * clean up deletion logic (#73) * fix artifacts location and merge syntax (#72) * add static ip (#75) * add static ip * global addresses apparently dont work * stop using git sha (#74) * stop using git sha * also deploy * ci cloudtools (#76) * Generalize deploy and fixes (#77) * fix deletion logic * generalize deploy secrets * add cloudtools * update deployment * print the ref name when describing prs to build next (#78) * fix big error (#79) * update deployment (#80) * add bgen-changes non-deploy, set 0.1 to deploy (#81) * more logging (#84) * more logging * update deployment * deploy cloudtools (#82) * deploy cloudtools * Update deployment.yaml * capitalization on Nealelab (#86) * update deployment (#87) * heal watched refs, not just those with active PRs (#85) In particular, try to deploy any watched ref * fix image (#88) * Fix and add test for review condensation (#89) In particular, we incorrectly let COMMENTED overwrite an APPROVE simply because it was more recent. * added gcr-push secret for hail-is/batch (#93) * added gcr-push secret for hail-is/batch * make batch deployable * update deployment (#95) update Dockerfile to handle new batch file structure * Quiet down gsutil copies at the end of build logs (#98) * add index (#100) * add CORS (#99) - fix the Makefile - add CORS package dependency - add CORS to all endpoints The CI does not serve secret information so accessing it from arbitrary domains is not a problem. * Generate temporary repos to test against (#97) * wip * a little closer * bugs & posix * fixes * setup endpoints * give notice of setup-endpoints.sh` * wait more than one minute when there are no k8s nodes available it takes about 60 seconds to start one * bad rebase, should have deleted this line * pretty sure ssl is not supported in ci2 * stop using ci-test repo and use better temp dir * hide token in test-locally * be professional * address a few comments * hide TOKEN * move to subdir (#105) * move to subdir * last few things moved
danking
pushed a commit
that referenced
this issue
Sep 25, 2018
* initial revision * wip * added deployment updated logging * added service * wip * wip * cancellation, primitive client library (unused) bidict jupyter deployment * batches, higher level api * fixed readme * minor * added tests, reorg * tests run in itself * wip added attributes removed job names create batch objects added callback (untested) * added callbacks, callback test * makefile changes * wip try to get callback test working on k8s cluster * tweaks * callback test works on k8s * starting to play with containerized spark some cleanup * Create setup.py * move setup.py to correct location * ignore compiled pyc files * describe how to use minikube with local docker images * add special note about `imagePullPolicy` * add an environment yaml * add a getting started section * add a dockerignore file * do not fail if callback fails * expose pod/container volumes * fix volumeMounts field name * add resources * add tolerations * add jobs listing * add jobs test * fix the environment * update Batch.create_job for all the new parameters * fix missing id * avoid crashing on bad event types * update dockerignore * retry event loops * Update server.py * fix dockerignore * fix dockerignore * stash the attributes sent by api * cache the status * fixed tests (#28) * Pr builder image (#27) * added pr-builder image logic * added missing files * address comments * add hail-ci-build.sh (#30) * add hail-ci-build.sh wip * wip * make hail-ci script work added shutdown endpoint * fix * don't log the (pod) log (#29) * added bash to pr-builder image (#32) * Add git (#35) * test * added git to pr-builder image * add python alias in pr-builder image * added curl to pr-builder image * Job delete (#36) * wip: added job/delete need to test * added delete, various * tag executable images (#38) * update deployment (#39) * make batch single threaded (#40) * make batch single threaded polling and k8s watch thread request into main thread in addition to k8s notifications, periodically poll k8s state * address comments * label batch job pods (#41) * add hail-ci-deploy.sh (#42) * added hail-ci-deploy.sh add docker to pr-builder image * added missing deployment.yaml.in removed deployment.yaml * consistency * addessed comments * install kubectl in image (#44) `gcloud components install kubectl` is failing with a non-obvious error message so I installed it directly * make batch subproject (#46) * make batch subproject fixed more deployment bugs * updated build image * fix test race condition job can complete before it is cancelled * authenticate docker to push to gcr.io (#48) * authenticate docker to push to gcr.io * add gcloud quiet (-q) * remove errant tab * logging to match ci * do callback asynchronously * do callback asynchronously * expose pod name as well (#49) * restart ci on deploy (#50) * Fix SHA check (#51) * Update hail-ci-deploy.sh * Update deployment.yaml.in * Update hail-ci-deploy.sh * Update hail-ci-deploy.sh * add /jobs/<id>/log endpoint (#52) * add /jobs/<id>/log endpoint added to api, client return completed job logs (including deleted ones) return in-progress logs (if there are any) return 404 (not empty) if there no logs to be found status['log'] returns the same log * fixed bug * fixed typo * prep to merge into monorepo clean up unused files, experiments (jupyter, spark, etc.) * updated build image * fixes hail-ci-build.sh handles all known projects fix batch to deploy only when changed * fixed typo * fixed README.md conflict. * activate environment * add cloudtools to list of project-changed.py projects
tpoterba
pushed a commit
to tpoterba/hail
that referenced
this issue
Feb 12, 2019
Made the os.mkdir statements safer and free from race conditions
daniel-goldstein
referenced
this issue
in daniel-goldstein/hail
Feb 3, 2022
Add Artifact Registry permissions for the batch-agent service account
danking
added a commit
to danking/hail
that referenced
this issue
Oct 11, 2023
Consider this:
```scala
class Foo {
def bar(): (Long, Long) = (3, 4)
def destructure(): Unit = {
val (x, y) = bar()
}
def accessors(): Unit = {
val zz = bar()
val x = zz._1
val y = zz._2
}
}
```
These should be exactly equivalent, right? There's no way Scala would compile the match into
something horrible. Right? Right?
```
public void destructure();
Code:
0: aload_0
1: invokevirtual hail-is#27 // Method bar:()Lscala/Tuple2;
4: astore_3
5: aload_3
6: ifnull 35
9: aload_3
10: invokevirtual hail-is#33 // Method scala/Tuple2._1$mcJ$sp:()J
13: lstore 4
15: aload_3
16: invokevirtual hail-is#36 // Method scala/Tuple2._2$mcJ$sp:()J
19: lstore 6
21: new #13 // class scala/Tuple2$mcJJ$sp
24: dup
25: lload 4
27: lload 6
29: invokespecial hail-is#21 // Method scala/Tuple2$mcJJ$sp."<init>":(JJ)V
32: goto 47
35: goto 38
38: new hail-is#38 // class scala/MatchError
41: dup
42: aload_3
43: invokespecial hail-is#41 // Method scala/MatchError."<init>":(Ljava/lang/Object;)V
46: athrow
47: astore_2
48: aload_2
49: invokevirtual hail-is#33 // Method scala/Tuple2._1$mcJ$sp:()J
52: lstore 8
54: aload_2
55: invokevirtual hail-is#36 // Method scala/Tuple2._2$mcJ$sp:()J
58: lstore 10
60: return
public void accessors();
Code:
0: aload_0
1: invokevirtual hail-is#27 // Method bar:()Lscala/Tuple2;
4: astore_1
5: aload_1
6: invokevirtual hail-is#33 // Method scala/Tuple2._1$mcJ$sp:()J
9: lstore_2
10: aload_1
11: invokevirtual hail-is#36 // Method scala/Tuple2._2$mcJ$sp:()J
14: lstore 4
16: return
```
Yeah, so, it extracts the first and second elements of the primitive-specialized tuple, constructs
a `(java.lang.Long, java.lang.Long)` Tuple, then does the match on that.
sigh.
danking
added a commit
that referenced
this issue
Oct 17, 2023
…13794) Consider this: ```scala class Foo { def bar(): (Long, Long) = (3, 4) def destructure(): Unit = { val (x, y) = bar() } def accessors(): Unit = { val zz = bar() val x = zz._1 val y = zz._2 } } ```  These should be exactly equivalent, right? There's no way Scala would compile the match into something horrible. Right? Right? ``` public void destructure(); Code: 0: aload_0 1: invokevirtual #27 // Method bar:()Lscala/Tuple2; 4: astore_3 5: aload_3 6: ifnull 35 9: aload_3 10: invokevirtual #33 // Method scala/Tuple2._1$mcJ$sp:()J 13: lstore 4 15: aload_3 16: invokevirtual #36 // Method scala/Tuple2._2$mcJ$sp:()J 19: lstore 6 21: new #13 // class scala/Tuple2$mcJJ$sp 24: dup 25: lload 4 27: lload 6 29: invokespecial #21 // Method scala/Tuple2$mcJJ$sp."<init>":(JJ)V 32: goto 47 35: goto 38 38: new #38 // class scala/MatchError 41: dup 42: aload_3 43: invokespecial #41 // Method scala/MatchError."<init>":(Ljava/lang/Object;)V 46: athrow 47: astore_2 48: aload_2 49: invokevirtual #33 // Method scala/Tuple2._1$mcJ$sp:()J 52: lstore 8 54: aload_2 55: invokevirtual #36 // Method scala/Tuple2._2$mcJ$sp:()J 58: lstore 10 60: return public void accessors(); Code: 0: aload_0 1: invokevirtual #27 // Method bar:()Lscala/Tuple2; 4: astore_1 5: aload_1 6: invokevirtual #33 // Method scala/Tuple2._1$mcJ$sp:()J 9: lstore_2 10: aload_1 11: invokevirtual #36 // Method scala/Tuple2._2$mcJ$sp:()J 14: lstore 4 16: return ``` Yeah, so, it extracts the first and second elements of the primitive-specialized tuple, ~~constructs a `(java.lang.Long, java.lang.Long)` Tuple~~ constructs another primitive-specialized tuple (for no reason???), then does the match on that. sigh.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From @cseed on September 24, 2015 16:26
Copied from original issue: cseed/hail#56
The text was updated successfully, but these errors were encountered: