Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error: multiple rlib candidates for kafka found #20

Closed
Lingling7 opened this issue Oct 1, 2015 · 13 comments
Closed

error: multiple rlib candidates for kafka found #20

Lingling7 opened this issue Oct 1, 2015 · 13 comments

Comments

@Lingling7
Copy link

I'm a beginner of kafka and rust, to use rust client as a consumer, I add the github example in src/main.rs

extern crate kafka;
use kafka::client::KafkaClient;
fn main() {
    let mut client = KafkaClient::new(&vec!("localhost:9092".to_string()));
    client.load_metadata_all();
    // OR
    // client.load_metadata(&vec!("my-topic".to_string())); // Loads metadata for vector of topics
 }

and add kafka = "*" in the cargo.toml dependencies

then I run $ cargo build I have the following error, is there anyone can help me to solve this?

   Compiling kafka v0.1.6 (file:///Users/admin/myrustdemo/kafka-rust-master)
src/main.rs:2:1: 2:20 error: multiple rlib candidates for `kafka` found
src/main.rs:2 extern crate kafka;
              ^~~~~~~~~~~~~~~~~~~
src/main.rs:2:1: 2:20 note: candidate #1: /Users/admin/myrustdemo/kafka-rust-master/target/debug/libkafka.rlib
src/main.rs:2 extern crate kafka;
              ^~~~~~~~~~~~~~~~~~~
src/main.rs:2:1: 2:20 note: candidate #2: /Users/admin/myrustdemo/kafka-rust-master/target/debug/deps/libkafka-73c7ae9da4235a7d.rlib
src/main.rs:2 extern crate kafka;
              ^~~~~~~~~~~~~~~~~~~
src/main.rs:2:1: 2:20 error: can't find crate for `kafka`
src/main.rs:2 extern crate kafka;
              ^~~~~~~~~~~~~~~~~~~
error: aborting due to 2 previous errors
Could not compile `kafka`.
@spicavigo
Copy link
Collaborator

I think you added kafka dependency to kafka library itself. i.e. you added kafka="*" to the cargo.toml of kafka-rust library.

To test the library out, you should

  1. cd /Users/admin/myrustdemo
  2. cargo new kafkademo --bin
  3. cd kafkademo
  4. cp ../kafka-rust-master/src/main.rs src/
  5. Add kafa dep to cargo.toml
  6. cargo build

Let me know if this doesn't work

@Lingling7
Copy link
Author

Thank you so much! I followed all your steps, and add [dependencies] kafka = "*" in cargo.toml of kafkademo, I ran into this error

    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling rustc-serialize v0.3.16
   Compiling winapi-build v0.1.1
   Compiling winapi v0.2.4
   Compiling byteorder v0.3.13
   Compiling libc v0.1.10
   Compiling gcc v0.3.17
   Compiling advapi32-sys v0.1.2
   Compiling rand v0.3.11
   Compiling miniz-sys v0.1.6
   Compiling flate2 v0.2.9
   Compiling num v0.1.27
   Compiling kafka v0.1.6
   Compiling kafkademo v0.1.0 (file:///Users/admin/myrustdemo/kafkademo)
src/main.rs:5:39: 5:74 error: mismatched types:
 expected `collections::vec::Vec<collections::string::String>`,
    found `&collections::vec::Vec<collections::string::String>`
(expected struct `collections::vec::Vec`,
    found &-ptr) [E0308]
src/main.rs:5     let mut client = KafkaClient::new(&vec!("localhost:9092".to_string()));
                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/main.rs:5:39: 5:74 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `kafkademo`.

is that means I need to delete &? (As what I saw in soulfly's post)
but I ran into a new question

   Compiling kafkademo v0.1.0 (file:///Users/admin/myrustdemo/kafkademo)
src/main.rs:6:5: 6:32 warning: unused result which must be used, #[warn(unused_must_use)] on by default
src/main.rs:6     client.load_metadata_all();
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: linking with `cc` failed: exit code: 1
note: "cc" "-m64" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "/Users/admin/myrustdemo/kafkademo/target/debug/kafkademo.0.o" "-o" "/Users/admin/myrustdemo/kafkademo/target/debug/kafkademo" "-Wl,-force_load,/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libmorestack.a" "-Wl,-dead_strip" "-nodefaultlibs" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libkafka-73c7ae9da4235a7d.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libflate2-98f57e2f7af2d432.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libnum-397f282c1d72fe58.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/librustc_serialize-7ff5bfc027146194.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/librand-bdfcc55d3466feb3.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libminiz_sys-fa48ce1a538dca79.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/liblibc-144c435538abd757.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libbyteorder-3e27c88aa235985f.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcollections-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_unicode-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librand-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liblibc-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcore-198068b3.rlib" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug/deps" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug/build/miniz-sys-fa48ce1a538dca79/out" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "-L" "/Users/admin/myrustdemo/kafkademo/.rust/lib/x86_64-apple-darwin" "-L" "/Users/admin/myrustdemo/kafkademo/lib/x86_64-apple-darwin" "-l" "snappy" "-l" "c" "-l" "m" "-l" "System" "-l" "pthread" "-l" "c" "-l" "m" "-l" "compiler-rt"
note: ld: warning: directory not found for option '-L/Users/admin/myrustdemo/kafkademo/.rust/lib/x86_64-apple-darwin'
ld: warning: directory not found for option '-L/Users/admin/myrustdemo/kafkademo/lib/x86_64-apple-darwin'
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error
Could not compile `kafkademo`.

@xitep
Copy link
Collaborator

xitep commented Oct 1, 2015

hello,

yes ... that & is currently a misleading point in the documentation (see #18) and will be resolved soon (i hope).

regarding the linking error: i think you'll need to have the native snappy library installed (don't know how to do this on windows, but on unix like systems it'd expect you to have /usr/lib/libsnappy.so.1.1.3 or similar).

@Lingling7
Copy link
Author

It works, thanks!!!

@Lingling7
Copy link
Author

this is what I put in main.rs now

extern crate kafka;
use kafka::utils;
use kafka::client::KafkaClient;
fn main() {
    let mut client = KafkaClient::new(vec!("localhost:9092".to_string()));
    //client.load_metadata_all();
    // OR
    client.load_metadata(vec!("rocksdbtest".to_string())); // Loads metadata for vector of topics
    let con = kafka::consumer::Consumer::new(client, "test-group".to_string(), "rocksdbtest".to_string())
             .partition(0);
    for msg in con {
        println!("{:?}", msg);
    }
 }

this is the warning when I type cargo build

src/main.rs:4 use kafka::utils;
                  ^~~~~~~~~~~~
src/main.rs:9:5: 9:59 warning: unused result which must be used, #[warn(unused_must_use)] on by default
src/main.rs:9     client.load_metadata(vec!("rocksdbtest".to_string())); // Loads metadata for vector of topics
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I have .exe now, but I cannot see anything when I ran cargo run, are there anything I miss?

This is the tree of kafademo

.
├── Cargo.lock
├── Cargo.toml
├── src
│   ├── consumer.rs
│   └── main.rs
└── target
    └── debug
        ├── build
        │   ├── advapi32-sys-cfef7a1f30f1e5f6
        │   │   ├── build-script-build
        │   │   ├── build_script_build.dSYM
        │   │   │   └── Contents
        │   │   │       ├── Info.plist
        │   │   │       └── Resources
        │   │   │           └── DWARF
        │   │   │               └── build_script_build
        │   │   ├── out
        │   │   └── output
        │   └── miniz-sys-fa48ce1a538dca79
        │       ├── build-script-build
        │       ├── build_script_build.dSYM
        │       │   └── Contents
        │       │       ├── Info.plist
        │       │       └── Resources
        │       │           └── DWARF
        │       │               └── build_script_build
        │       ├── out
        │       │   ├── libminiz.a
        │       │   └── miniz.o
        │       └── output
        ├── deps
        │   ├── libadvapi32-cfef7a1f30f1e5f6.rlib
        │   ├── libbuild-304afb6bdff23d72.rlib
        │   ├── libbyteorder-3e27c88aa235985f.rlib
        │   ├── libflate2-98f57e2f7af2d432.rlib
        │   ├── libgcc-f53b41fe21770ed8.rlib
        │   ├── libkafka-73c7ae9da4235a7d.rlib
        │   ├── liblibc-144c435538abd757.rlib
        │   ├── libminiz_sys-fa48ce1a538dca79.rlib
        │   ├── libnum-397f282c1d72fe58.rlib
        │   ├── librand-bdfcc55d3466feb3.rlib
        │   ├── librustc_serialize-7ff5bfc027146194.rlib
        │   └── libwinapi-21b078e9a1931364.rlib
        ├── examples
        ├── kafkademo
        ├── kafkademo.dSYM
        │   └── Contents
        │       ├── Info.plist
        │       └── Resources
        │           └── DWARF
        │               └── kafkademo
        └── native

@xitep
Copy link
Collaborator

xitep commented Oct 2, 2015

i think we need to improve the documentation a little bit. in your particular case, however ...

a)
... you can get rid of the warnings by 1) deleting that unused use kafka::utils and 2) capturing the result of the call to client.load_metadata. The quick and dirty way could be just something like let _ = client.load_metadata(...) (here you merely capture the result into a variable named _ and that makes the compiler happy). The correct way to deal with the result value would be to see whether load_metadata method resulted in an error or not. A good resource of this topic (error handling) is surely the rust book and Andrew Gallant's "Error Handling in Rust".

b)
regarding the fact that your program doesn't print anything, i would suspect that the topic rocksdbtest does not exist in your kafka broker. the consumer silently skips requests for non-existing topics. the topic is the third argument in your call to kafka::consumer::Consumer::new. I would suggest to verify it exists.

@Lingling7
Copy link
Author

Thanks for the very detail info, xitep!!!
I can use fetch_messages and print out now!!!^_^
But still cannot use the consumer, probably the "test-group", I use the default name in consumer.properties.java "test-consumer-group".
This is just test on the local, and I only build one topic "test"

extern crate kafka;
use kafka::client::KafkaClient;
use kafka::consumer::Consumer;
//use kafka::utils;
fn main() {
    // 1. Connect to Kafka
    let mut client = KafkaClient::new(vec!("localhost:9092".to_owned()));

    let kafka_metadata =client.load_metadata(vec!("test".to_owned())); 


    // client.load_metadata(vec!("my-topic".to_owned())); // 

    // 2. fetch messages: single (Topic, Partition, Offset)
    let offsetmsg= client.fetch_messages("test".to_owned(), 0, 0);
    println!("offsetmsg: {:?}", offsetmsg);
    //3. consumer

    let con = Consumer::new(client, "test-consumer-group".to_owned(), "test".to_owned()).partition(0);
    for msgs in con {
       println!("{:?}", msgs);
    }
}

This is partial result

offsetmsg: Ok([TopicMessage { topic: "test", partition: 0, offset: 0, error: None, message: [91] }, TopicMessage { topic: "test", partition: 0, offset: 1, error: None, message: [9, 123] },...TopicMessage { topic: "test", partition: 0, offset: 155, error: None, message: [93] }])

No result form the consumer.

@spicavigo
Copy link
Collaborator

@Lingling7 Could you fetch the offsets for this consumer and verify that its less than the greatest offset for the partition you are consuming?

@Lingling7
Copy link
Author

@spicavigo Do you mean check the offset numbers (id:0-155 ) with the consumed messages numbers(as below, 156)?

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Consumed 156 messages

In addition, the info in "messages: [91] " (mylast comment) is not the message in kafka, messages here looks like the metadata.

@spicavigo
Copy link
Collaborator

@Lingling7
Copy link
Author

@spicavigo thanks, I add this in main.rs

let resp = client.fetch_group_topic_offset("test-consumer-group".to_owned(),"test".to_owned());
    println!("resp: {:?}", resp);

result like this
resp: Ok([TopicPartitionOffsetError { topic: "test", partition: 0, offset: -1, error: Some(UnknownTopicOrPartition) }])

@Lingling7
Copy link
Author

I can use the consumer now, because I didn't set the offesetkey to the earliest, so I used

bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliest config/consumer.properties test

But how can I set this through rust, should I set from the consumer?

For the messages form consumer, I just got the numbers, how can I get the true text messages?

msgs: TopicMessage { topic: "test", partition: 0, offset: 1, error: None, message: [32, 32, 32, 32, 125] }
msgs: TopicMessage { topic: "test", partition: 0, offset: 2, error: None, message: [109, 115, 103, 116, 101, 115, 116, 95, 50] }

And if I want to load the data to rocksdb, should I use fetch or consumer?

@Lingling7
Copy link
Author

The numbers in message is vector of bytes (utf-8), I need to transfer them to string

use std::str

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants