Can not update collection UnknownUpdateOperation(...)

[peepo5]

collection
    .update(
        Some(&mk_document! { "_key": key_name }),
        &mk_document! { "value": value },
    )
    .unwrap();

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnknownUpdateOperation("value")', src/db.rs:143:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I want to update the entry with the "_key" as key_name's value field, however, it displays the above error. any help?

[vincentdchan]

You need to specific the update operation, so try:

collection
    .update(
        Some(&mk_document! { "_key": key_name }),
        &mk_document! {
          "$set": &mk_document! {
             "value": value
          }
       },
    )

Other operation can be found at docs/en-US/Update.md

[peepo5]

error[E0277]: the trait bound `Value: From<&polodb_bson::Document>` is not satisfied
   --> src/db.rs:140:6
    |
140 |                   &mk_document! {
    |  __________________^
141 | |                    "$set": &mk_document! {
142 | |                       "value": value
143 | |                    }
144 | |                 },
    | |_________________^ the trait `From<&polodb_bson::Document>` is not implemented for `Value`
    

This is what I get after using identical code as yours.

[vincentdchan]

A little mistake:

collection
    .update(
        Some(&mk_document! { "_key": key_name }),
        &mk_document! {
          "$set": mk_document! {    // <- do NOT need '&'
             "value": value
          }
       },
    )

[peepo5]

👍 testing now

[peepo5]

Alright that works! As I said in my other issue, It would be nice to have some examples, even just copy paste code to get a better understanding of this module in use. Thanks for the help.

[peepo5]

I compared the database values before and after update and it seems nothing changed. Very weird.
I ran my program twice at once, checking the db while in function.
Before update:

[
    Trade {
        ...
        main_id: 88877584825,
        sl_id: None,
        tp_id: None,
    },
]

After update (nothing changed):

[
    Trade {
        ...
        main_id: 88877584825,
        sl_id: None,
        tp_id: None,
    },
]

After function finished:

[
    Trade {
        _id: Some(
            310723,
        ),
        timestamp_open: 1634630287,
        filled: true,
        risk: 1,
        main_id: 88877584825,
        sl_id: Some(
            90149687,
        ),
        tp_id: Some(
            90140261,
        ),
    },
]

So this seems working.

Then after however, I tried to insert another entry into the database:

thread 'main' panicked at 'index 1 is greater than length 1', /home/admin/.cargo/registry/src/github.com-1ecc6299db9ec823/polodb_core-0.10.0/page/data_page_wrapper.rs:77:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

To insert, I am using the following function:

collection
    .insert(&mut mk_document! {
	"_id": id,
	"timestamp_open": td.timestamp_open.to_string(),
	"filled": td.filled,
        "risk": td.risk.to_string(),
	"main_id": td.main_id,
	"sl_id": if td.sl_id == None {0} else {td.sl_id.unwrap()},
	"tp_id": if td.tp_id == None {0} else {td.tp_id.unwrap()},
    })
.unwrap();

Do I need to specify the insert type to avoid this?

Also on another try, for some reason, when using collection.find_all(...), it gave me an empty result (when it shouldnt) and then it gave me a similar error.

"testa" command runs this function:

pub fn db_get_ftrades() -> Result<Vec<Trade>, Error> {
	let mut db = Database::open(database_location().as_str()).unwrap();
	let mut collection = db.collection("ftrades").unwrap();
	let all_trades = collection.find_all().unwrap();
	let mut trade_array: Vec<Trade> = Vec::with_capacity(all_trades.len());
	
	for doc in all_trades {
		let sl_id = doc.get("sl_id").unwrap().unwrap_int();
		let tp_id = doc.get("tp_id").unwrap().unwrap_int();
		trade_array.push(
			Trade {
				_id: Some(doc.get("_id").unwrap().unwrap_int().to_string().parse::<Decimal>()?),
				timestamp_open: /*DateTime::parse_from_str(*/doc.get("timestamp_open").unwrap().unwrap_string().parse::<Decimal>()?/*, "%s")?*/,
				filled: doc.get("filled").unwrap().unwrap_boolean(),
				risk: doc.get("risk").unwrap().unwrap_string().parse::<Decimal>()?,
				main_id: doc.get("main_id").unwrap().unwrap_int() as u64,
				sl_id: if sl_id == 0 {None} else {Some(sl_id as u64)},
				tp_id: if tp_id == 0 {None} else {Some(tp_id as u64)},
			}
		);
	}

	Ok(trade_array)
}

It errors on the db.collection("ftrades").unwrap() for reference

[vincentdchan]

This error demonstrates that your db has been corrupted. Could you give me a minimal program or procedure descriptions to reproduce？

[peepo5]

I will soon.

[peepo5]

Just a question, do you need to update the database using a command before using the database again in the same function?

[vincentdchan]

Are you saying using collection again or using database again?

It's designed to be allow using collection for multiple times.
Database is designed to be used multiple times too, but limited. Only one instance can read/write database at the same time. Otherwise, the thread will be locked.

[vincentdchan]

But corruption is unexpected, the database should be safe.

[peepo5]

https://github.com/termcrypt/termcrypt/blob/master/src/ftx_exchange/ftx_inter.rs (line 247-576)
https://github.com/termcrypt/termcrypt/blob/master/src/db.rs

These are the relevant files.
I managed to replicate the issue on a test project.

cargo init then change main.rs:

//this is a temporary file to test databases within termcrypt.
use anyhow::{
    Error,
    //bail
    Result,
};
use chrono::{Local};
use polodb_bson::mk_document;
use polodb_core::Database;
use rand::Rng;
use rust_decimal::prelude::*;
use rust_decimal_macros::dec;
use rustyline::Editor;

#[derive(Debug)]
pub struct Trade {
    pub _id: Option<Decimal>,
    pub timestamp_open: Decimal,
    pub filled: bool,
    pub risk: Decimal,
    pub main_id: u64,
    pub sl_id: Option<u64>,
    pub tp_id: Option<u64>,
}

fn main() {
    let mut line_main = Editor::<()>::new();
    loop {
        let read_line = line_main.readline("> ");

        match read_line.unwrap().as_str() {
            "o" => {
                new_order().unwrap();
            }
            "g" => {
                println!("{:#?}", db_get_ftrades().unwrap())
            }
            _ => {}
        }
        println!();
    }
}

pub fn database_location() -> String {
    format!("database")
}

pub fn new_order() -> Result<(), Error> {
    let mut db = Database::open(database_location().as_str()).unwrap();
    let mut collection = db.collection("ftrades").unwrap();

    let mut line_main = Editor::<()>::new();

    let main_db_id = db_insert_ftrade(Trade {
        _id: None,
        timestamp_open: Local::now().timestamp().to_string().parse::<Decimal>()?,
        filled: false,
        risk: dec!(1),
        main_id: rand::thread_rng().gen_range(10..9999999), //random number placeholder for real id
        sl_id: None,
        tp_id: None,
    })
    .unwrap();

    line_main.readline("Useless pause. Press enter.")?;
    collection
        .update(
            Some(&mk_document! { "_id": main_db_id }),
            &mk_document! {
                "$set": mk_document! {
                    "sl_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
        )
        .unwrap();
    line_main.readline("Useless pause. Press enter.")?;

    collection
        .update(
            Some(&mk_document! { "_id": main_db_id }),
            &mk_document! {
                "$set": mk_document! {
                    "tp_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
        )
        .unwrap();

    println!("done");
    Ok(())
}

pub fn db_insert_ftrade(td: Trade) -> Result<i64, Error> {
    let mut db = Database::open(database_location().as_str()).unwrap();
    let mut collection = db.collection("ftrades").unwrap();

    let all_trades = collection.find_all().unwrap();
    let mut id: i64;
    //makes sure id is not already in ftrades
    loop {
        id = rand::thread_rng().gen_range(10..999999);
        let mut failed = false;
        for trade in &all_trades {
            if id == trade.get("_id").unwrap().unwrap_int() {
                failed = true
            }
        }
        if !failed {
            break;
        }
    }

    collection
        .insert(&mut mk_document! {
            "_id": id,
            "timestamp_open": td.timestamp_open.to_string(),
            "filled": td.filled,
            "risk": td.risk.to_string(),
            "main_id": td.main_id,
            "sl_id": if td.sl_id == None {0} else {td.sl_id.unwrap()},
            "tp_id": if td.tp_id == None {0} else {td.tp_id.unwrap()},
        })
        .unwrap();
    Ok(id)
}

pub fn db_get_ftrades() -> Result<Vec<Trade>, Error> {
    let mut db = Database::open(database_location().as_str()).unwrap();
    let mut collection = db.collection("ftrades").unwrap();
    let all_trades = collection.find_all().unwrap();
    let mut trade_array: Vec<Trade> = Vec::with_capacity(all_trades.len());

    for doc in all_trades {
        let sl_id = doc.get("sl_id").unwrap().unwrap_int();
        let tp_id = doc.get("tp_id").unwrap().unwrap_int();
        trade_array.push(
			Trade {
				_id: Some(doc.get("_id").unwrap().unwrap_int().to_string().parse::<Decimal>()?),
				timestamp_open: /*DateTime::parse_from_str(*/doc.get("timestamp_open").unwrap().unwrap_string().parse::<Decimal>()?/*, "%s")?*/,
				filled: doc.get("filled").unwrap().unwrap_boolean(),
				risk: doc.get("risk").unwrap().unwrap_string().parse::<Decimal>()?,
				main_id: doc.get("main_id").unwrap().unwrap_int() as u64,
				sl_id: if sl_id == 0 {None} else {Some(sl_id as u64)},
				tp_id: if tp_id == 0 {None} else {Some(tp_id as u64)},
			}
		);
    }

    Ok(trade_array)
}

then change Cargo.toml:

[dependencies]
polodb_core = "0.10.0"
polodb_bson = "0.10.0"
rand = "0.8.4"
anyhow = "1.0.44"
rust_decimal = "1.16"
rust_decimal_macros = "1.16"
rustyline = "9.0.0"
chrono = "0.4"
dirs = "4.0"

then test like how i did in the image. I think the database works fine without the collection.update btw you can test removing both of the updates.

[vincentdchan]

Get it. I will try your program.

[peepo5]

Are you able to reproduce the issue?

[vincentdchan]

Not yet. Sorry I am busy these days, I will try tomorrow.

[peepo5]

👍 thank you.

[vincentdchan]

What toolchain are you using? I can not compile your proj with stable Rust. @peepopoggers

[peepo5]

1.55

[vincentdchan]

stable/nightly/beta ?

[peepo5]

stable.
try rustup default 1.55.0 or something

[vincentdchan]

I've reproduced successfully, I'll see what happened.

[peepo5]

Hi there. I do not want to rush you in any way, but I would like to know if this is a problem with my code's logic or your module. This problem is stunting development for my app and I may have to consider moving database modules if there is no answer soon.

[vincentdchan]

I've located the problem. It seems that you open two DB instances of one file in the same program space, the key is in the function new_order, reorder the codes, and the problem will be fixed:

pub fn new_order() -> Result<(), Error> {
        // insert data here, and close the database
	let main_db_id = db_insert_ftrade(Trade {
		_id: None,
		timestamp_open: Local::now().timestamp().to_string().parse::<Decimal>()?,
		filled: false,
		risk: dec!(1),
		main_id: rand::thread_rng().gen_range(10..9999999), //random number placeholder for real id
		sl_id: None,
		tp_id: None,
	})
		.unwrap();
        // reopen the database, so not conflicts
	let mut db = Database::open(database_location().as_str()).unwrap();
	let mut collection = db.collection("ftrades").unwrap();

	let mut line_main = Editor::<()>::new();

	line_main.readline("Useless pause. Press enter.")?;
	collection
		.update(
			Some(&mk_document! { "_id": main_db_id }),
			&mk_document! {
                "$set": mk_document! {
                    "sl_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
		)
		.unwrap();
	line_main.readline("Useless pause. Press enter.")?;

	collection
		.update(
			Some(&mk_document! { "_id": main_db_id }),
			&mk_document! {
                "$set": mk_document! {
                    "tp_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
		)
		.unwrap();

	println!("done");
	Ok(())
}

[vincentdchan]

Also, the deeper reason to this bug is that, two instances contain different caches of data. So PoloDB itself should have a mechanism to avoid cache conflicts. I will consider some method to avoid this problem. Thanks for your feedback.

[vincentdchan]

Also, I've read your code. You can update an item in one statement:
Original:

line_main.readline("Useless pause. Press enter.")?;
	collection
		.update(
			Some(&mk_document! { "_id": main_db_id }),
			&mk_document! {
                "$set": mk_document! {
                    "sl_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
		)
		.unwrap();
	line_main.readline("Useless pause. Press enter.")?;

	collection
		.update(
			Some(&mk_document! { "_id": main_db_id }),
			&mk_document! {
                "$set": mk_document! {
                    "tp_id": rand::thread_rng().gen_range(10..9999999)
                }
            },
		)
		.unwrap();

After:

line_main.readline("Useless pause. Press enter.")?;
	collection
		.update(
			Some(&mk_document! { "_id": main_db_id }),
			&mk_document! {
                "$set": mk_document! {
                    "sl_id": rand::thread_rng().gen_range(10..9999999),
                    "tp_id": rand::thread_rng().gen_range(10..9999999),
                }
            },
		)
		.unwrap();
	line_main.readline("Useless pause. Press enter.")?;

This is not neccessary, but better code.

[peepo5]

Thank you very much. I will impliment this later. You may also want to add a warning to the readme to not access a database in this way to prevent corruption.

[peepo5]

Just out of curiosity, is it an issue with database initiation or collection initiation?

[vincentdchan]

It's about database initiation.

[vincentdchan]

Use newer version will be more safe:

polodb_core = "0.10.2"
polodb_bson = "0.10.2"

[peepo5]

I have seen the warning for busy database while using it now, that is a great addition.