Run pipelines on LUH cluster #62
Comments
|
Cui is working on (1)
|
|
-> RandomPip is running |
|
;) It's also in our interest.... And here is the extended LUH cluster wikipage (the lower parts are mostly "dangerous"...) |
|
Great! |
|
Not sure, this is LUH decision. We had to sign some papers that we loose all our money if we download their data ;) Just kidding, I think we will take your pipeline and run it there for you reporting on the issues that we see. |
|
Adaptive switch over RandomPip is working. |
|
As a follow-up - LUH infrastructure is now working with coordination.commandCompletion.onEvent = true. Will become active with next restart. |
|
For (4)+(5) the financial data source may need an adjustment to read from the file system rather than HDFS (not available in LUH cluster). Either there is a way to convince Miroslav or we have to modify the TSI source. Initial discussion with Christoph regarding the Twitter source (works with file system) are ongoing to synchronize the work and the approach... |
|
But can't the Okeanos HDFS be also used? (It will be slower because of the internet but it could be an alternative at least for testing...) |
|
On Thu, 19 Jan 2017 03:12:32 -0800 Holger Eichelberger <notifications@github.com> wrote:
Hi Holger,
[storm@master02 ~]$ hadoop fs -put ./test.hdfs /data/storm/
[storm@master02 ~]$ hadoop fs -ls /data/storm/
Found 1 items
-rw-r--r-- 3 storm users 5 2017-01-19 13:57 /data/storm/test.hdfs
[storm@master02 ~]$ hadoop fs -cat /data/storm/test.hdfs test
Cheers,
miroslav
… For (4)+(5) the financial data source may need an adjustment to read
from the file system rather than HDFS (not available in LUH cluster).
Either there is a way to convince Miroslav or we have to modify the
TSI source. Initial discussion with Christoph regarding the Twitter
source (works with file system) are ongoing to synchronize the work
and the approach...
--
Dr. Miroslav Shaltev
- Systemadministration -
Forschungszentrum L3S
Appelstrasse 9a
30167 Hannover
Phone: +49 (0)511 762 - 17777
Fax: +49 (0)511 762 - 17779
E-Mail: shaltev@l3s.de
|
|
Cool. From our meeting I had in mind that we do not have access to HDFS. Then let's keep things simple, put data under /data/storm in HDFS and configure the infrastructure respectively. I will tell TSI... |
|
Just to let you know, today I spotted another attack at our HDFS; this time data ransom was included! According to this, these attacks are quite widespread. |
|
How bad, as the cluster of LUH is behind an external server, there the risk shall be lower. @L3SQualimaster any news on the FocusPip? |
|
Infrastructure setup has been changed to take up the LUH hdfs with base path /data/storm as suggested by Miroslav (thanks again). The respective setup information is being passed to the worker by the infrastructure via the DML configuration class ( It's now up to the sources to take up this information and to copy the required data into the HDFS. How is the time plan there? |
|
|
|
Fine :) |
|
BTW, the HDFS user/group setup is not done, so the infrastructure may complain about that while extracting the setup files for Stefan. But as far as I know, Tuan is trying to find a way how the Stakeholder applications could access the pipeline results, then we can also figure out whether extracting the setup files also holds/is needed in the LUH setup. |
|
... the installation needs a further classpath entry for HDFS. Discussing with Miroslav... |
|
Hi everyone, I got an error while trying to start the infrastructure on hadoop2: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration (I tried to run ./main.sh) |
|
That's exactly what my last post was about. Therefore, I need a change done by root. And please do not use main.sh on your cluster. There the infrastructure is installed as a service. I'll let you know as soon as the classpath entry is clarified with Miroslav (or you may call him directly ;)) |
|
Ok, fixed thanks to Miroslav. Updated the Wiki in this respect (main.sh). Started PriorityPip. Comes up and reports Here is the full trace hdfs.path, hdfs.url are set up. Simulation settings as discussed in #59 so far not. Are they needed. |
|
Thanks for fixing! The FocusPip starts now, but immediately crashes because of the FinancialSource which is not a surprise given that the financial data is not yet reachable. Btw, I would not use the PriorityPip for testing since it is very old and I think nobody is maintaining it. |
|
Ok (see trace above), how to set up the financial simulation source? @ap0n |
|
Apostolos: we can add the financial dataset here: /local/home/storm/datasets Then we just have have to adapt the Source. |
|
FYI, we just killed the pipeline TransferPip |
|
Btw, how can we see the status of a pipeline life cycle ? :) |
|
Just through the infrastructure logs :o |
|
We did not tweak the Storm UI... - or through QM-IConf (adaptation/runtime log). But therefore we also need to have the adaptation port available ;) |
|
... as long as the infrastructure logs "Elevating to".. the respective phase is not reached. For both critical phases (CREATED and INITIALIZED) it logs also which nodes are expected and which are missing. CREATED means that all nodes must have send an initial event via the event bus and that they are ready for receiving the initial algorithm assignment. If exceptions happen here, CREATED can easily be prevented. Then the adaptation issues commands/signals to set the initial algorithms. INITIALIZED means that all nodes have received and confirmed their initial algorithm and are ready for processing. If INITIALIZED is reached, the infrastructure connects data sinks and sources and processing shall start, i.e., it switches the pipeline to STARTED. |
|
Hi, what about the running FocusPip? I would like to test the Load shedding (just enabled on PriorityFinancialPip) and need an infrastructure restart (local model)... |
|
And from the FocusPip: node19: 2017-02-01T12:40:26.879+0100 b.s.util [ERROR] Async loop died! (PipelineVar_7_Source1) after some Got playerList parameter. New value: addmarketPlayer/0,1,2,3,4,5... |
|
Killing the FocusPip... |
|
Changing financial data set to overload scenario 2... |
|
@eichelbe I'm testing the HBase connection and need to change and recompiled DataManageLayer, and manually copied into provided_libs directory in LUH cluster. Do I have to re-run main.sh to reload the library? It seems the code is not updated when I restarted the pipelines .... |
|
Definitely it is not updated based on the pipelines as not included (provided libs). As stated above, in your cluster running main.sh is not the right way as its execution is controlled by a service manager. Ok, Jenkins is done. I'll do the update and the distribution and let you know... |
|
What about the running transfer pip? |
|
no I didn't commit it yet. Just thought we could hot test it quickly via manual copies. |
|
Okay just committed now |
|
Then it will take a while until we can update the infrastructure. Did not know this. Otherwise a patch and overriding the jar after a maven build would have worked... |
|
Jenkins is ready. Distributing libraries and restarting infrastructure on LUH cluster. |
|
model loaded, infrastructure is up. |
|
So, I hard coded the Hbase configuration instead of getting values from DataManagementConfiguration, just to see if the problem is about ZooKeeper or not. The good news now is we didn't get the "Connection refused" error, so as long as qm.infrastructure.cfg managed to be visible to the pipeline, I guess we are good to go. For the moment, we have the following error: 2017-02-01T18:55:59.683+0100 e.q.Configuration [ERROR] While reading configuration file /var/nfs/qm/qm.infrastructure.cfg: /var/nfs/qm/qm.infrastructure.cfg (No such file or directory) Not sure how this file is distributed to the infrastructure ... |
|
Which component is emitting that error? |
|
I actually didn't know which component causing this - my guess is somewhere in the EventManager that tries to locate the path of qm.infrastructure.cfg (I wasn't deeply aware of this actually) Here is an excerpt from the log when I ran FocusPip (so no ReplayMechanism around): 2017-02-02T10:55:51.916+0100 e.q.F.t.PipelineVar_7_Source1Source [INFO] Prepared--basesignalspout.... FocusPip/PipelineVar_7_Source1 |
|
The bolts and spouts are initializing the event manager as needed and this happens before the shown trace based on Storm conf information. They do not rely on that path. So something else may call that, as far as I remember some TSI components are doing this with fixed path as fallback, but this is just a guess. Of course, we may emit the stacktrace to find the cause, but this may be as confusing as the sinks indicating with a stack trace that the external service is not reachable. More importantly, do you get the correct hbase setup information from DML cfg or not without trying to initialize it manually... and when do you try to get this information. Please note that the correct information is available earliest after the super.prepare(...) call, not in any constructor (which is anyway the wrong place to do heavy-weight initializations here)). |
|
I will do some re-parallelization tests on LUH cluster now. May cause strange effects in other pipelines on correlation computation. It's probably better to not do further tests at the moment, probably until lunch. I'll let you know. |
|
I see that now the cluster is free. Could we run some tests now ? In addition, we constantly made some changes now in DML to spot the issues (just commenting / un-commenting some pieces of code to inspect). How can we restart the platform conveniently instead of emailing around? |
|
Partially. try running your tests now. the effects shall now be rather local on PrioFinPip. If you take the responsibility of fixing problems from an update until the infra is running again, please feel free to read the wiki and to use the respective scripts. |
|
Is the TransferPip still needed? Can I go on? |
|
Hi Holger, yes you can kill the pipelines and proceed now. We are scrutinizing locally now |
|
Hi, anyone using the LUH cluster by now. I would like to update/restart it and to give the TimeTravelPip another try (just having an idea where the strange exception could come from). |
|
Update/restart done... |
|
@eichelbe Hi Holger, could you tell us where to see the infrastructure log of the currently running TransferPip ? |
|
Hi Tuan, |
For testing and backup-plan
(1) Run RandomPip on LUH cluster (SUH)
(2) Replace storm to Adaptive Storm* and change worker configuration for larger pipelines (SUH)
(3) Run RandomPip on LUH cluster again (SUH)
(4) Run TransferPip on LUH cluster (LUH)
(5) Run FocusPip on LUH cluster (LUH)
(6) For TSI: Run Time Travel Pipeline for startup time debugging on LUH cluster (SUH + TSI)
(7) Rund TransferPip + FocusPip on LUH cluster
(8) Test QM-IConf connection into LUH cluster (requires SSH tunneling)
(9) Test Application connection into LUH cluster (requires SSH tunneling)
Please document process below!
After (3) we will communicate how to work with the cluster on the project Wiki!
*required for adaptation and for identifying timing issues (extension over storm, can be further extended if needed)
The text was updated successfully, but these errors were encountered: