Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
IrcLog2008 12 03
17:18:19 * stevenknight (n=stevenkn@nat/google/x-ff6fd1a26f0686e6) has joined #scons 17:25:16 * Jason_ (n=[Jason@bementil-116.illinois.prairieinet.net](mailto:Jason@bementil-116.illinois.prairieinet.net)) has joined #scons 17:29:26 * garyo-home (n=[email@example.com](mailto:firstname.lastname@example.org)) has joined #scons 17:31:26 * [GregoryNoel](GregoryNoel) is here 17:31:38 <[GregoryNoel](GregoryNoel)> Hi, guys, are we ready? 17:31:51 <garyo-home> Hi Greg. Ready as I'll be. :-/ 17:32:23 <[GregoryNoel](GregoryNoel)> I only see limited comments in the spreadsheet; it's gonna be a slow night. 17:32:53 <garyo-home> yes, sorry. Just doing some now. 17:33:02 <stevenknight> hey 17:33:07 <stevenknight> me too 17:34:15 <[GregoryNoel](GregoryNoel)> 1945? 17:34:42 <garyo-home> steven to research & pick one option. 17:34:47 <garyo-home> w/ Ludwig. 17:34:57 <garyo-home> IMHO. 17:35:25 <[GregoryNoel](GregoryNoel)> I'll go with that, at least for now. We need to come up with something better in the long run. 17:36:17 <garyo-home> Not sure about your comment re: rewriting implicit dependency logic, but won't go there now. 17:36:44 <stevenknight> agree w/what you guys are saying 17:37:17 <[GregoryNoel](GregoryNoel)> I think we can do more than what we are now by caching negatives, but it's not the time to redesign it. 17:37:36 <garyo-home> right. 17:38:00 <[GregoryNoel](GregoryNoel)> so research, Steven + Ludwig? 17:38:11 <garyo-home> good. 17:38:19 <[GregoryNoel](GregoryNoel)> And once the decision is reached, Ludwig to implement? 17:38:28 <stevenknight> done 17:38:31 <[GregoryNoel](GregoryNoel)> done 17:38:35 <[GregoryNoel](GregoryNoel)> 2258 17:38:36 <garyo-home> 2258: stevenknight to add a hook to get help text, 2.x p4? 17:39:07 <[GregoryNoel](GregoryNoel)> Hmmmm..... 17:40:06 <[GregoryNoel](GregoryNoel)> Yeah, something like that. His suggestion is awkward, but it should be possible to work out something. 17:40:38 <[GregoryNoel](GregoryNoel)> And for 2.x p4, we don't need to volunteer anyone. 17:40:41 <[GregoryNoel](GregoryNoel)> yet 17:40:48 <garyo-home> agree. 17:40:52 <stevenknight> okay, i'm back from the spreadsheet 17:41:04 <stevenknight> 2258: done 17:41:09 <stevenknight> 2.x p4, future draft pick 17:41:10 <[GregoryNoel](GregoryNoel)> 2261 17:41:38 <stevenknight> it's a google guy, so i'd like a crack at it 17:41:58 <[GregoryNoel](GregoryNoel)> We have another issue looking at this already; one should be closed. 17:42:00 <garyo-home> Shouldn't he be using the target filename as the target? (And not rm/mkdir either) 17:41:59 <stevenknight> if i can't come up with a real case, i'll mark it invalid 17:42:35 <garyo-home> ok, that's good enough for me. Steven to find testcase or mark invalid. 17:43:05 <[GregoryNoel](GregoryNoel)> or dup to the other one 17:43:16 <stevenknight> yeah, i guess i view it similar to unpacking a .tar.gz into a directory 17:43:22 <stevenknight> i think you should be able to specify the directory 17:43:30 <stevenknight> and have it do the right thing w.r.t. changes to the source(s) 17:43:41 <stevenknight> but i'll mark it invalid if I'm just dreaming 17:43:47 <[GregoryNoel](GregoryNoel)> done 17:44:00 <[GregoryNoel](GregoryNoel)> 2262 17:44:12 <stevenknight> i think it should be trivial 17:45:26 <[GregoryNoel](GregoryNoel)> If you think it's trivial, I'll let you have it. 17:45:20 <stevenknight> 2262: anytime, anyone 17:45:29 <[GregoryNoel](GregoryNoel)> anytime is fine 17:45:39 <garyo-home> The OP says he'd like "not .py" but I think .py makes the most sense. 17:45:41 <stevenknight> okay, 2262 anytime stevenknight 17:46:15 <Jason_> I like the .scons ,myself 17:46:17 <stevenknight> while we're at it, i'd go for both .py and .scons 17:46:36 <stevenknight> and draw the line there 17:46:38 <[GregoryNoel](GregoryNoel)> I've also seen .sc used. 17:46:45 <garyo-home> ok, why not. Just pick an order, seems fine to me. 17:47:04 <garyo-home> Greg: I think .sc is too short. 17:47:17 <[GregoryNoel](GregoryNoel)> Sigh. Seems too complex; I'd still wontfix it as I indicated. 17:47:02 <stevenknight> isn't .sc for [SuperCalc](SuperCalc) files...? 17:47:05 * stevenknight shows his age... 17:47:12 <garyo-home> [SuperCalc](SuperCalc)?!! 17:47:38 <[GregoryNoel](GregoryNoel)> Yeah, I remember it, too. 17:47:46 * [GregoryNoel](GregoryNoel) _really_ shows his age. 17:47:47 <stevenknight> 2262: anytime, stevenknight 17:47:52 <[GregoryNoel](GregoryNoel)> done 17:48:03 <[GregoryNoel](GregoryNoel)> 2264 17:48:27 <garyo-home> Greg, you looked at this most -- is the order reversal the only thing going on here? 17:48:28 <stevenknight> 2.x p2 gregnoel? 17:49:16 <[GregoryNoel](GregoryNoel)> No, the dependencies aren't reversed, reversing the order of the names on the command line produces different trees. 17:49:43 <garyo-home> ok, I believe you :-) 17:49:47 <[GregoryNoel](GregoryNoel)> No matter which way you list the names on the command line, it should show the same dependencies. 17:49:57 <garyo-home> do you mind taking it? 17:50:45 <[GregoryNoel](GregoryNoel)> I know almost nothing about that part of the code, so I'd rather not, but if you push, I'll consider it. 17:51:04 <stevenknight> 2.x p2 can be future draft pick 17:51:16 <stevenknight> or else put my name on it 17:51:16 <[GregoryNoel](GregoryNoel)> I'll go with that 17:51:27 <stevenknight> okay, 2.x p2 future draft pick 17:51:28 <stevenknight> done 17:51:40 <[GregoryNoel](GregoryNoel)> 2265 17:51:50 <garyo-home> ah, yes. 17:51:56 <Jason_> just so it is known this is not new 17:52:15 <stevenknight> 2265: 1.2 p1 stevenknight 17:52:16 <garyo-home> Jason_, you mean 2265? 17:52:23 <Jason_> Yes 17:52:27 <stevenknight> like I said, I'm just in Taskmaster lately 17:52:28 <Jason_> I filed it 17:52:30 <garyo-home> thanks, Steven. 17:52:47 <garyo-home> Ah, you're *that* Jason -- welcome! 17:52:55 <Jason_> Thanks 17:53:13 <[GregoryNoel](GregoryNoel)> stevenknight, done 17:53:19 <[GregoryNoel](GregoryNoel)> and thanks 17:53:18 <stevenknight> Jason_: glad to have you here 17:53:24 <Jason_> hope we can talk latter :-) 17:53:38 <stevenknight> yep 17:53:46 <stevenknight> i have a few non-issue topics i'd like to discuss as well 17:53:54 <garyo-home> ok 17:54:12 <[GregoryNoel](GregoryNoel)> 2266: I'll let you guys fight it out 17:54:17 <stevenknight> 2266: 1.x p3 stevenknight 17:54:36 <Jason_> we have seen this .. it depends on the python used 17:54:43 <garyo-home> 2266: not sure why this doesn't work for him. I do this all the time. What python is he using? 17:54:58 <garyo-home> Do NOT use cygwin python. 17:55:15 <Jason_> That follows what we see here as well 17:55:29 <Jason_> cygwin python will mess up a windows based build 17:55:37 <garyo-home> 2266: let me take it and suggest that to him. 17:55:43 <[GregoryNoel](GregoryNoel)> done 17:55:54 <[GregoryNoel](GregoryNoel)> 2267 17:56:47 <stevenknight> either 1.2 and defer if investigation shows we can live with it 17:56:56 <[GregoryNoel](GregoryNoel)> This is the new Action.py code I put in, but I can't see how it's not initialized. 17:56:57 <stevenknight> or resesarch with the option of making it a 1.2 blocker 17:58:19 <[GregoryNoel](GregoryNoel)> Seeing no hands, I'll research it. It's a bad week for that, but I'll try to make time. 17:58:22 <garyo-home> Greg: possible exception issue? 17:58:27 <[GregoryNoel](GregoryNoel)> garyo-home, huh? 17:58:35 <garyo-home> never mind. 17:59:15 <[GregoryNoel](GregoryNoel)> so 2267 research me? 17:59:20 <stevenknight> done 17:59:47 <stevenknight> [GregoryNoel](GregoryNoel): let me know if I can help, I've had to debug things like this many times 17:59:53 <[GregoryNoel](GregoryNoel)> thanks 18:00:05 <[GregoryNoel](GregoryNoel)> Before we go on, can I insert a topic? 18:00:17 <garyo-home> sure 18:00:34 <stevenknight> sure, let's do GN's first, then Jason_, then me 18:00:30 <[GregoryNoel](GregoryNoel)> I'd like to change the agenda slightly. I'd like to add a permanent item to discuss the current status of the release schedule and how we're doing on it. 18:00:30 <[GregoryNoel](GregoryNoel)> Discussion points would be what's expected in the next release(s) (whether checkpoint, candidate, or public), adjustments to the schedule, whether we should do any retriage for current or future releases, things like that. 18:00:30 <[GregoryNoel](GregoryNoel)> I'd put it between the current issues and the backlog. (Aside: we're almost to the home stretch for finishing off the backlog (hopefully we'll kill off 2005 this time or next time); after that, the time now spent on backlog will be taken up by bickering about retriage of issues, so we should discuss the release schedule strategy before the tactics. This positioning puts it in the right place for the future.) 18:01:13 <stevenknight> +1 to discussing release schedule regularly 18:01:24 <garyo-home> greg: +1 to your agenda change. I always review the schedule before the meeting anyway. 18:02:16 <[GregoryNoel](GregoryNoel)> OK, I'll do it. Since the slot would be now, should we discuss the schedule now? 18:02:21 <stevenknight> yes 18:02:23 <garyo-home> yes. 18:02:31 <[GregoryNoel](GregoryNoel)> Schedule: 1.2 was due out 24 November. Needless to say, we didn't make it. And issues 2265 and 2267 could be a blockers; we may need to slip 1.2 until that's resolved. 18:02:31 <[GregoryNoel](GregoryNoel)> We were supposed to have a checkpoint out 1 December. We didn't make that, either. 18:02:31 <[GregoryNoel](GregoryNoel)> Assuming we get the release out this week, we have some decisions to make. Should we slide the rest of the schedule by a week, so the next checkpoint is 8 December? Cancel the 1 December checkpoint and keep the current schedule, with the next checkpoint 15 December? Or something else? 18:02:31 <[GregoryNoel](GregoryNoel)> I lean toward dropping the 1 December checkpoint, but I could be easily persuaded differently. 18:02:59 <stevenknight> i put out a release candidate 11/25? 18:03:33 <[GregoryNoel](GregoryNoel)> I think that's about right 18:03:28 <garyo-home> I have three 1.2 tickets on my plate. None serious except maybe #2048 which I'm working on but fails in weird ways 18:04:42 <stevenknight> oh, wait, I see--the release schedule is 12/1 checkpoint that was supposed to be *after* 1.2 was out 18:04:45 <stevenknight> and i'm late with 1.2 18:04:51 <stevenknight> right? 18:05:10 <garyo-home> I think so. I'd like to get 1.2 out soon then put vs_revamp in for next checkpoint after that. 18:05:40 <[GregoryNoel](GregoryNoel)> stevenknight, yes 18:05:37 <stevenknight> right, and I actually have batched Visual C/C++ compilation working on a private branch that I'd like to go in 18:05:56 <garyo-home> wow! 18:05:57 <Jason_> I would like to add stuff to the revamp 18:06:12 <garyo-home> Jason: good. You have an intelc.py version of it too I think. 18:06:12 <stevenknight> okay, so let's review 1.2 issues and push anything that's not a potential/likely blocker in 1.3 18:06:18 <Jason_> We have some work not finished that will improve it and make it faster 18:06:38 <garyo-home> OK. Let's get it into the trunk, then review & apply your changes 18:06:46 <Jason_> yep... still working on that part 18:06:48 <garyo-home> ok? 18:07:02 <Jason_> plus we have early support for vs 2010 and intel 11 18:07:02 <stevenknight> garyo_home: "it" into the trunk: "it" == vs_revamp? 18:07:14 <stevenknight> Jason_: sweet 18:07:31 <garyo-home> steven: yes, it==vs_revamp. 18:07:36 <Jason_> I have "people" that know "people" 18:07:38 <stevenknight> okay, agreed 18:08:01 <Jason_> Well my fixes to the revamp.. are a bit different from the work David did 18:08:02 <garyo-home> There are 40 open 1.2 issues. 18:08:05 <[GregoryNoel](GregoryNoel)> Almost 40 issues scheduled for 1.2 still 18:08:13 <[GregoryNoel](GregoryNoel)> garyo-home, jinx 18:08:47 <garyo-home> Jason_: send a summary of your ideas & differences to the list, we'll discuss it there. 18:08:55 <Jason_> ok 18:09:13 <garyo-home> It's a bigger forum, David will be present, etc. 18:09:16 <stevenknight> a lot of the 40 issues are readily rollable to 1.3 18:09:19 <stevenknight> doc issues, e.g. 18:09:57 <[GregoryNoel](GregoryNoel)> Mine are all doc and testing (I haven't figured out how to test Action._subproc yet) 18:10:05 <garyo-home> 15 P2s, the rest P3 & P4. 18:10:49 <stevenknight> oh, and 40 doesn't count the two potential blockers we discussed tonight 18:11:15 <stevenknight> quick, quick glance suggests all P3 & P4 are automatically deferrable 18:12:07 <garyo-home> agreed. 18:12:09 <stevenknight> and I think the cournape P2 issues are because we once thought vs_revamp was going to make it into 1.2 18:12:54 <garyo-home> Is Benoit around? Any likelihood of his 4 issues getting done? 18:13:06 <stevenknight> slim, he's been AWOL for a while 18:13:21 <garyo-home> thought so. 18:13:31 <stevenknight> worth considering reassigning his issues so they don't languish 18:14:04 <garyo-home> Yes. I suggest moving to 1.3 for now, then reassigning. 18:15:02 <[GregoryNoel](GregoryNoel)> garyo-home, agree 18:15:13 <stevenknight> okay, so it sounds like the only two issues that don't go to 1.3 are the two new potential blockers 18:15:32 <[GregoryNoel](GregoryNoel)> I can do that with a mass move. 18:15:41 <stevenknight> [GregoryNoel](GregoryNoel): thanks 18:15:33 <stevenknight> we get patches for those (or research suggests deferring to 1.3) and then I cut another 1.2 candidate 18:16:02 <garyo-home> What's up w/ 2249 though? 18:16:44 <stevenknight> dunno... 18:16:47 <garyo-home> (Not that it should be in 1.2 anyway, but how'd it get there?) 18:17:07 <stevenknight> vs_revamp, we thought it would be in 1.2 18:17:08 <garyo-home> It can move to 1.3 too since it's vs_revamp related. 18:17:59 <garyo-home> ok, so given all that, what's the proposed 1.2 release date? 18:18:10 <[GregoryNoel](GregoryNoel)> ASAP? 18:18:13 <stevenknight> stake in the ground: 12 December, a week from Friday 18:18:20 <stevenknight> candidate 5 December, two days 18:18:32 <stevenknight> for the two potential blockers 18:18:41 <[GregoryNoel](GregoryNoel)> works; I'll update the roadmap 18:18:44 <garyo-home> ok, so I have some time to sneak in some little things before the candidate, then we freeze, right? 18:19:00 <stevenknight> yep 18:19:04 <garyo-home> good. 18:19:20 <stevenknight> okay, shall we move on to Jason_'s topic? 18:19:37 <garyo-home> yes. 18:19:57 <garyo-home> I looked a little at the doc for Parts; it is quite serious! 18:20:22 <Jason_> Well we have some new tool coming out... it has to be a bit serious 18:20:41 <garyo-home> Jason, I wonder if some of it couldn't be done with SCons Tools instead, but basically it looks useful. 18:20:49 <garyo-home> What's the new Intel tool coming out? 18:21:05 <Jason_> well I have a lot to say on all of this 18:21:21 <Jason_> Idea the doc i sent is about extending Scons... 18:21:31 <garyo-home> The basic idea is a way to make a scalable composition of libs etc., right? 18:21:32 <Jason_> here i think tools needs many improvements 18:22:05 <Jason_> well yes... but also about making large project sain to deal with 18:22:33 <garyo-home> Right, by imposing certain conventions that make it subdividable nicely. 18:22:36 <Jason_> having a unified build system is very important 18:22:55 <Jason_> but also trying to be flexible 18:23:07 <stevenknight> sorry, I haven't had time to look before now 18:23:09 <Jason_> it is a hard balance... 18:23:12 <garyo-home> Of course; lots of people have large unified build systems with SCons. Your way is one way of formalizing a structure. 18:23:18 <stevenknight> is the doc the .7z file you attached? 18:23:29 <garyo-home> Steven: yes, it's in there. 18:23:37 <Jason_> yes.. the problem with the team i work with is that we are all over the place 18:23:48 <Jason_> having a "make system" does not work well 18:23:56 <Jason_> yes 18:24:16 <stevenknight> agh, i don't have 7zip on this system 18:24:24 <Jason_> it is in word 2007 format.. I can give you an update in something else like html 18:25:25 <Jason_> so the first question for me is how can we best stare this code 18:25:54 <Jason_> My boss would like this to be part of SCons instead of it own project... 18:26:27 <Jason_> however it might be best to have it as a seperate project with heavy SCons involment 18:27:11 <garyo-home> I've long thought it would be good for SCons to have a contrib/ subtree, so that might be one way to go. The downside is you're tied to SCons release schedule and other stuff though. Separate is probably easier. 18:27:36 <Jason_> well we have our own svn branch here at Intel 18:27:48 <Jason_> ideally we would try to move to a public one 18:28:15 <Jason_> but we have some stuff in a subdir call "pieces" that i cannot ship 18:28:33 <garyo-home> Not sure what the best way to host a project like that is, but there are many places to look. 18:28:44 <Jason_> as it is internal add on that have knowleage of internal Intel networks 18:28:59 <garyo-home> You'll have to make it work "out of the box" from the public repo, of course. 18:29:10 <Jason_> yep that is the idea 18:29:22 <Jason_> the sample in the code i gave you should work out of the box 18:29:33 <garyo-home> I think everyone will be happier if it is separate from scons but closely related, at least for now. 18:29:37 <Jason_> the full sample will need svn to be installed :-) 18:30:09 <garyo-home> But we should spend some time at least reading the doc to get an idea of the scope and goals of the project. 18:30:17 <Jason_> If this agreed, I will start the process to make a tigris Parts project 18:30:39 <garyo-home> How about calling it SConsParts or something to show its relation to SCons? 18:30:58 <Jason_> that is fine with me 18:31:12 <Jason_> any other votes? 18:31:58 <garyo-home> Let's talk about the tech details and direction more at the next bug party meeting in 2 weeks. 18:32:11 <stevenknight> sorry, still trying to open things up 18:32:28 <stevenknight> got the .7z open, but i don't have Office on the system I switched to... 18:32:28 <Jason_> in 2 week I will be in florida *not* working 18:32:31 <garyo-home> btw, is this going to appear in a public tool from Intel? 18:32:35 <Jason_> so after that is fine 18:33:01 <garyo-home> Jason: ok, whenever you are around. But give us a week or so to look it over. 18:33:10 <Jason_> sure 18:33:14 <garyo-home> Esp. given 1.2 release schedule :-) 18:33:29 <stevenknight> Jason_: there's an early effort here at Google to do something that sounds similar to Parts 18:33:38 <stevenknight> but it looks like you're much farther along 18:33:46 <stevenknight> when I look, I 18:33:47 <Jason_> so the parts stuff will be public.. MIT under Intel with the intent to be move as much as possible to SCons 18:34:02 <Jason_> you work at Google 18:34:14 <stevenknight> I'm going to look for areas of possible cooperation 18:34:27 <stevenknight> yes, almost 11 months now 18:34:38 <Jason_> oh .. I thought it was VMware 18:34:45 <stevenknight> that was before, and on contract 18:34:56 <Jason_> well you should contact me... INtel and google have a lot going on 18:35:18 <stevenknight> the problem you're addressing is what we're running into too 18:35:46 <Jason_> so are many projects at Intel .... 18:35:56 <stevenknight> right, SCons is pretty good low-level plumbing 18:36:08 <Jason_> very powerful stuff however 18:36:15 <stevenknight> but there's no consistency for higher layer plug-and-play between components 18:36:35 <stevenknight> esp. in the open source world it all depends on how the person wrote the SCons config 18:36:37 <Jason_> I think as i under scons better and you parts.. I think there is a lot of value here to improve on 18:36:47 <stevenknight> agreed 18:37:10 <Jason_> the plug and play in very important 18:37:30 <stevenknight> right, it's the real value add of Autotools (IMHO) 18:37:31 <Jason_> when I was at a start up before intel.. one person controled the make scripts 18:37:58 <stevenknight> being able to approach any package and know how to at least build it consistently 18:38:12 <Jason_> in there larger cross geo.. developer fight over details and perfection 18:38:14 <stevenknight> sounds like Parts goes even further though 18:38:23 <Jason_> so everything breaks . 18:38:26 <stevenknight> right 18:38:32 <stevenknight> so following up on what Gary said 18:38:42 <Jason_> cross geo poltics i have found can make it hard to get simple stuff done 18:38:56 <stevenknight> I'm going to take a closer look after 1.2 is out 18:39:03 <Jason_> ok 18:39:24 <stevenknight> and perhaps touch base with you off line to compare first impressions 18:39:26 <Jason_> I will send out friday an update to the code and document ( small tweaks and fixes) 18:39:33 <stevenknight> thanks 18:39:41 <Jason_> so FYI 18:39:41 <stevenknight> if you can convert doc to some non-Word format that'd help me 18:39:51 <garyo-home> same here 18:39:49 <Jason_> WW 51 i am out of town 18:40:05 <garyo-home> What is "WW 51"? 18:40:08 <Jason_> I will be around other wise and will keep track of the e-mails 18:40:23 <Jason_> week of Dec 15 18:40:34 <garyo-home> ah yes. 18:40:43 <stevenknight> okay, i'll look for your update and we'll go from there 18:40:50 <Jason_> ok 18:40:57 <Jason_> thank you for your time 18:41:12 <garyo-home> ok, Steven I think it's your turn now! 18:41:17 <stevenknight> ok 18:41:21 <garyo-home> Jason_: yr welcome. 18:41:40 <stevenknight> one is the Visual C/C++ batch builder change I mentioned 18:41:58 <stevenknight> took a while, but it's close to solid 18:42:09 <stevenknight> some unit tests need attention 18:42:27 <garyo-home> Impressive, can't wait to try it. We have a lot of vc++ compiled into a few big libs. 18:42:54 <stevenknight> i'm going to try to get it in right away after 1.2 is out so it gets soak time 18:43:16 <stevenknight> it has some changes to the Taskmaster that aren't terribly extensive 18:43:23 <stevenknight> but may have unintended side effects 18:43:31 <[GregoryNoel](GregoryNoel)> (BRB. My wife just arrived with food and I've got to eat it while it's hot.) 18:43:56 <garyo-home> Do users have to do anything or does it just work? 18:43:59 <stevenknight> darn, bad timing, I was going to ask Greg something about Taskmaster NG 18:44:03 <Jason_> If you need someone to help test the taskmaster on a large project we are happy to help 18:44:24 <stevenknight> it's controlled by setting MSVC_BATCH=True (or 1 or whatever) 18:44:59 <garyo-home> sounds great. Does it generalize to other batch-buildable things? 18:45:06 <stevenknight> yes 18:45:10 <[GregoryNoel](GregoryNoel)> (I'm still reading, but I have to put down the burger to type.) 18:45:18 <stevenknight> okay, cool 18:45:21 <Jason_> if you can give me the detail of what to do i will give it a run on a few new i7 systems 18:45:41 <stevenknight> the Taskmaster issues is that this suggests the Taskmaster is handling the wrong granularity of input 18:45:45 <stevenknight> it's looking at individual Nodes 18:45:55 <stevenknight> and I'm beginning to think it should be looking at Executors 18:46:13 <stevenknight> [GregoryNoel](GregoryNoel), just chew on that a little and let me know if it sounds crazy 18:46:16 <stevenknight> based on your TNG work 18:46:37 <stevenknight> you can do your own batch builder at the Action() level 18:46:51 <stevenknight> simplest case is Action('$FOOCOM', batch_key=True) 18:47:02 <[GregoryNoel](GregoryNoel)> That's what TNG does: the transverse graph is in (a wrapper around) the Executor. 18:47:14 <stevenknight> sweet, so I'm not entirely crazy, at least 18:47:21 <stevenknight> maybe this helps move us toward TNG 18:47:52 <stevenknight> batch_key=True will separate all calls to the given Builder using that action 18:48:59 <stevenknight> into grouped batches for the four-tuple (Action, env) 18:49:05 <stevenknight> excuse me, two-tuple 18:49:22 <stevenknight> batch_key can also be a function that takes (action, env, target, source) 18:49:39 <stevenknight> and returns the key to be used for separating batches 18:50:37 <stevenknight> so the MSVC_BATCH support uses the four-tuple (action, env, target.dir, source.dir) as the key 18:50:56 <garyo-home> That's very clever. 18:51:05 <stevenknight> with the upshot that you get batched compilation of all object files in a given directory 18:51:26 <stevenknight> because $CPPPPATH can affect your #includes based on the directory 18:51:36 <garyo-home> ... and the same compiler options etc. 18:51:49 <stevenknight> right, out of the same env 18:52:14 <garyo-home> Is it a lot faster? 18:52:31 <stevenknight> for our config (Google Chrome) only about 10%-15% 18:52:41 <stevenknight> which is nothing to sneeze at, but still, we were hoping for more 18:52:50 <garyo-home> That's nothing to sneeze at (jinx) 18:53:04 <stevenknight> on the other hand, i haven't gone in and profiled and optimizd it yet 18:53:17 <stevenknight> i'll probably upload to a subversion branch so people can try it out early 18:53:20 <garyo-home> For us, we use -O3 with the Intel compiler so all the time goes into the link step, but I'm sure this will help us even in that case. 18:53:21 <stevenknight> and send out email+doc 18:53:50 <stevenknight> (i've been doing this work experimenting with mercurial as a front end private branch development) 18:54:07 <stevenknight> okay, enough of that 18:54:24 <garyo-home> what else? 18:54:34 <stevenknight> next: Google contributors to SCons 18:54:41 <Jason_> so i have a question 18:54:49 <stevenknight> yes? 18:55:03 <Jason_> is there any plans to improve the depend checker for C/C++ 18:55:16 <Jason_> there is a bit of an issue with BOOST headers for use 18:55:21 <stevenknight> i.e. make it like a real C preprocessor? 18:55:34 <Jason_> We have a work around... but it seem to cause more trouble than it solves 18:55:39 <stevenknight> there's an experimental module that will do that 18:56:08 <Jason_> well one that is smart enought to get the #include FOOBAR stuff 18:56:16 <stevenknight> yes 18:56:27 <stevenknight> take a look at (IIRC) Scanner/C.py 18:56:31 <Jason_> how can we try it out 18:56:59 <Jason_> branch in SCons? 18:57:01 <stevenknight> there's some commented out code that uses a cpp.py module to do enough "real" CPP stuff to handle boost 18:57:05 <Jason_> or in the main code 18:57:09 <stevenknight> no, it's checked in trunk 18:57:33 <stevenknight> just hasn't been productized by giving it the right user-configurable hook 18:57:43 <stevenknight> if you want to finish that piece and contribute it back, it'd be very welcome 18:58:04 <stevenknight> next: google contributors 18:58:30 <stevenknight> i have more people here starting to look at SCons internals 18:58:39 <stevenknight> especially w.r.t. performance 18:59:08 <stevenknight> which is good 18:59:23 <stevenknight> but we're finding that a lot of the flexibility we prize in SCons 18:59:53 <stevenknight> is antithetical to the kind of caching we'd need to do to really speed up things 19:00:08 <stevenknight> i'm wrestling with how best to channel this interest and energy 19:00:21 <garyo-home> that's hard. 19:00:34 <stevenknight> in ways that help SCons overall as well as Google's own purposes 19:00:50 <stevenknight> a Google fork of SCons is unattractive 19:00:58 <garyo-home> Right, certainly. 19:01:12 <garyo-home> Some kind of "mode"? 19:01:17 <stevenknight> but taking some of this work back into main line runs serious risk of breaking backwards compatibility 19:02:05 <garyo-home> and also making some existing idioms just plain not work, possibly. 19:02:12 <stevenknight> right 19:02:31 <garyo-home> How invasive is it? 19:02:57 <stevenknight> well, we're in early stages 19:03:07 <stevenknight> so more hypthetical than not 19:03:12 <garyo-home> ok. 19:03:27 <stevenknight> one patch i have queued tries to avoid starving child worker threads 19:03:39 <stevenknight> (actually doesn't try, it does avoid it) 19:03:57 <stevenknight> by just feeding as many ready Tasks as possible into the queue 19:04:00 <stevenknight> that doesn't break compatibility 19:04:20 <garyo-home> That seems great. In general, I think if it's really valuable, breaking simple back compat is ok (i.e. people have to do some simple recoding), but forcing larger changes is problematic. 19:04:23 <stevenknight> but it's in that pesky Taskmaster area where unintended affects abound 19:04:25 <[GregoryNoel](GregoryNoel)> Sounds like TNG 19:05:00 <stevenknight> mm, maybe I should send this patch to you to see what you think 19:05:36 <stevenknight> we were able to do some visualization (using Incredibuild) to see the starvation at work without the patch, and no starvation with it 19:06:32 <[GregoryNoel](GregoryNoel)> Sure, I can take a look. You should also look at TNG. 19:06:33 <stevenknight> tradeoff is more CPU cycles in the master thread searching the whole DAG for work more frequently 19:06:55 <[GregoryNoel](GregoryNoel)> TNG avoids that. 19:07:09 <stevenknight> okay, let's swap code and see how close we are 19:07:33 <stevenknight> are there issues holding up TNG? 19:08:02 <[GregoryNoel](GregoryNoel)> No, not really. Just lack of time. 19:08:08 <stevenknight> okay 19:08:12 <stevenknight> i'll send you the patch from here 19:08:37 <stevenknight> so now that I'm articulating this 19:08:38 <[GregoryNoel](GregoryNoel)> (here?) 19:08:43 <stevenknight> Google 19:09:30 <stevenknight> the meta issue is that this could bring a lot more impactive changes quickly into SCons 19:09:42 <stevenknight> with attendant possibility for destabilizing things 19:10:04 <stevenknight> in order to not hold up stuff, I'm going to have to spend (more) work time actually, oh, integrating patches 19:10:07 <[GregoryNoel](GregoryNoel)> (my fingers are still sticky, but I'm otherwise back.) 19:10:08 <stevenknight> in general, not just from Google 19:10:34 <stevenknight> and to do that I'm having to draft people here to cover some of the front-line work i've been trying to do on the Google Chrome build 19:11:12 <stevenknight> one other possibility I considered was to have a "google branch" in our SVN repository 19:11:29 <stevenknight> and pluck changes from that into trunk 19:11:32 <[GregoryNoel](GregoryNoel)> opposed: branches by features, not source 19:11:49 <stevenknight> yeah, that makes sense 19:12:23 <stevenknight> okay, so i just need to become johnny-on-the-spot with integration 19:12:33 <stevenknight> and maybe recruit additional committers / integrators 19:12:40 <stevenknight> from Google or elsewhere 19:12:50 <garyo-home> can you do that from Google? Would be useful. 19:13:05 <garyo-home> We haven't had much luck from the users list to date. 19:13:20 <stevenknight> can I do...? get more people from here working on SCons-the-project? 19:13:57 <garyo-home> Yes. 19:14:18 <stevenknight> there's certainly no institutional barrier 19:14:57 <garyo-home> Maybe get another googler or two on the dev list, tag along to a bug party... 19:15:01 <stevenknight> client-side projects here are getting behind SCons pretty heavily 19:15:09 <[GregoryNoel](GregoryNoel)> Doesn't Google have a rule that you can spend up to 20% of your time "doing good"? Recruit in that space. 19:15:43 <stevenknight> that still ends up being driven by whether people have an itch to scratch 19:15:51 <stevenknight> that they want to spend their 20% on 19:16:07 <garyo-home> but SCons is *sooo* *coool*! 19:16:07 <[GregoryNoel](GregoryNoel)> Of course. But fixing their build system to support themselves better is a strong itch. 19:16:08 <stevenknight> the potential itch here is speeding up builds for their projects 19:16:31 <garyo-home> That makes sense. 19:16:53 <Jason_> you need faster startup times? 19:17:01 <stevenknight> faster null build times 19:17:14 <garyo-home> steven: agreed, same here. 19:17:18 <[GregoryNoel](GregoryNoel)> ditto 19:17:24 <garyo-home> hence your caching ideas. 19:17:28 <Jason_> ya... big issue here.. I have a work around... but it a hack 19:17:37 <stevenknight> Jason_: what's your workaround? 19:17:48 <Jason_> we have a DB file we make 19:18:03 <Jason_> we use this to figure out if we need to define soemthing to SCons 19:18:13 <Jason_> this speeds up builds a great deal 19:18:26 <Jason_> we woudl have NULL build of 8-10 minutes 19:18:39 <stevenknight> so you only conditionally load parts of the configuration based on this DB? 19:18:47 <Jason_> got it down to 13-63 19:18:53 <Jason_> depending on the target 19:19:06 <Jason_> 13-63 secs 19:19:22 <Jason_> yes... but we have to have one good run 19:19:33 <stevenknight> right, to prep the DB with legitimate info 19:19:35 <stevenknight> that makes sense 19:19:39 <Jason_> else we don't know what is defined 19:20:01 <stevenknight> kind of related to this is making the DAG in SCons a real DAG 19:20:08 <stevenknight> (something I know Greg is very much in favor of) 19:20:18 <stevenknight> with separate tuples representing the edges 19:20:19 <Jason_> Same here 19:20:35 <[GregoryNoel](GregoryNoel)> Er, I thing you mean making the .sconsign a real DAG, but yes, I do. 19:20:42 <Jason_> then you can throw some well known task processing code at it 19:20:53 <Jason_> such as stuff used by a compiler 19:20:59 <stevenknight> right 19:20:59 <garyo-home> That scares me a little unless you explicitly allow runtime modification of the DAG 19:21:16 <[GregoryNoel](GregoryNoel)> white papers on request. 19:21:27 <stevenknight> and be able to infer what to build from known changed files 19:21:43 <garyo-home> ... but maybe users have to explicitly identify when they're modifying the DAG at runtime to invalidate and/or rescan 19:21:48 <stevenknight> instead of always having to walk from targets down, searching for what might have changed 19:21:57 <stevenknight> [GregoryNoel](GregoryNoel): request, send white papers 19:22:19 <stevenknight> garyo-home: you have a use case in mind? 19:22:31 <garyo-home> Dynamic source generators. 19:22:42 <Jason_> Agreeded 19:22:46 <garyo-home> Where you don't know all the generated source names up front. 19:22:48 <Jason_> we have that as well 19:22:51 <stevenknight> that should still be possible 19:23:05 <garyo-home> Good, I rely heavily on that. 19:23:21 <stevenknight> i think the nodes would still have .sources, .implicit, .explicit etc. 19:23:23 <Jason_> a simple example if a Doxygen builder 19:23:31 <[GregoryNoel](GregoryNoel)> In point of fact, I'd like to optimize the null case first, when the SConscripts haven't been changed, and work from there. 19:23:33 <stevenknight> but instead of pointing directly to ohter Nodes 19:23:40 <Jason_> it can't really know what it will build up front 19:23:41 <stevenknight> they'd be lists of DAG edges 19:24:00 <stevenknight> [GregoryNoel](GregoryNoel): agree in general 19:24:03 <garyo-home> Jason_: a very good use case there. 19:24:18 <stevenknight> but how do we know when the null case is optimized "enough" ? 19:24:30 <garyo-home> Steven: I have to think about that before I understand you. 19:24:34 <[GregoryNoel](GregoryNoel)> Zero time in parsing. 19:24:50 <stevenknight> wow, configurations are so different 19:25:00 <stevenknight> parsing is a really small part of our null build time 19:25:32 <[GregoryNoel](GregoryNoel)> I wonder... 19:26:02 <stevenknight> profiling shows we're heavily dominated by string (re-)expansion 19:26:46 <[GregoryNoel](GregoryNoel)> True, and working the bug list yesterday I found a case where a string was expanded *six* times to the same effect. 19:26:35 <Jason_> idea... you might want to look at the parts mapper.py file 19:26:59 <Jason_> we sort of replace expantions.. to get around the time issue here 19:27:07 <stevenknight> Jason_; aha 19:27:27 <stevenknight> I'll look at what you did as a source of ideas 19:27:40 <stevenknight> one of my big problems is I've spent so much time focusing on making corner cases work 19:27:44 <Jason_> part of this is me hacking a wrapper around the scanner object to force a few expansions 19:27:59 <stevenknight> that I'm prone to ignore effective solutions that really help the common case(s) 19:28:03 <Jason_> 5-10% difference in speed 19:28:32 <stevenknight> we got a pretty good improvement just running big long CPPPATH lists through env.subst() when assigning them 19:28:58 <Jason_> the problem was i did not see it when you added it :-( 19:29:13 <stevenknight> ? not in SCons itself, in our configuration 19:29:32 <Jason_> oh... never was able to use them 19:30:29 <stevenknight> anyone else about ready to turn into a pumpkin too? 19:30:36 <stevenknight> I should head for home... 19:30:39 <garyo-home> yes, time for me to go. 19:30:41 <[GregoryNoel](GregoryNoel)> Long past time for me 19:30:48 <stevenknight> okay, many thanks guys 19:30:52 <garyo-home> Thanks a lot as usual, guys! 19:30:54 <[GregoryNoel](GregoryNoel)> two weeks? 19:30:59 <garyo-home> ok for me. 19:31:00 <stevenknight> hopefully we'll see some increased activity coming up 19:31:06 <stevenknight> two weeks is good for me 19:31:07 <Jason_> latter! 19:31:12 <[GregoryNoel](GregoryNoel)> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed? 19:31:12 <[GregoryNoel](GregoryNoel)> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed? 19:31:12 <[GregoryNoel](GregoryNoel)> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed? 19:31:13 <stevenknight> should have 1.2 out by then... 19:31:27 <[GregoryNoel](GregoryNoel)> oops... 19:31:26 <stevenknight> what I tell you three times is true... 19:31:29 <garyo-home> greg: yes, yes, yes. 19:32:04 <stevenknight> for the snark *was* a boojum, you see... 19:32:08 <garyo-home> :-) 19:32:17 <[GregoryNoel](GregoryNoel)> snicker-snack 19:32:17 <garyo-home> ok, g'night all. 19:32:22 * Jason_ has quit ("Leaving") 19:32:25 <[GregoryNoel](GregoryNoel)> cul 19:32:29 <stevenknight> later 19:32:39 * garyo-home has quit ("[ChatZilla](ChatZilla) 0.9.84 [Firefox 3.0.4/2008102920]") 19:47:01 * stevenknight has quit ("Leaving")
Clone this wiki locally
Press h to open a hovercard with more details.